Let’s say we have keyspaces test1 and test2 with tables testtable1,testtable2, and testtable3 respectively which we want to make a backup of. In order to do that, we need to send a backup operation to one of the sidecars.

Let’s say that we have three nodes, where hostname is node1,node2, and node3 respectively and it resolves to IP.

You need to send the following JSON as body for POST http://node1:4567/operations with application/json Content-Type and Accept headers.

Firstly, we said that this operation is of type backup—each operation has to have this field with its own type.

Next, we specify our bucket to backup the cluster to via storageLocation—we do not expose credentials in this request. For example, the interaction with S3 has to be backed by some credentials. Please refer to Esop documentation to learn how the credentials resolution for each supported cloud environment works.

Thirdly, we specify snapshotTag under which this backup will be taken and which we need to refer to upon restoration.

Additionally, we specify entities that tells us what keyspaces to actually backup. Icarus does not backup anything you do not specify, so if you want to backup system_schema where table definitions are you have to do that yourself. If you want to backup just one table (or only tables), you need to use ks1.t1,ks2.t2 format.

Last but not least, we say that this request is global by setting globalRequest: true. Once this request is sent to node1 Icarus, that instance will act as a coordinator of the cluster-wide backup. If we did not specify it, there would be a backup of just that node1 done.

After submission of this operation, we may check what operations there are by calling GET http://node1:4567/operations (again JSON Accept).

We see that our Icarus running on node node1 (Cassandra runs there too, having same IP/hostname) is running two backup operations. There are also other fields returned which the reader can consult by reading our Swagger spec. The first fields returned are the same for every operation (type, id, creationTime, state, errors, progress, startTime).

The last operation is the global one (globalRequest:true). Internally, Icarus detected what the cluster topology is and it executed other, individual, backup request, to each node in a cluster. This fact is reflected in having two backup operations. If you check closely, you see that globalOperation for the first operation is set to false and storageLocation is updated with name of a cluster, data center, and node id.

The similar response to the first operation would be found for the other two nodes too—node2 and node3, (with specific fields updated to reflect each node).

Once an operation is finished or changes its state, we might see it from state field. Here, we see that such operation is COMPLETED. An operation bumps over these states:

• submitted—operation is submitted to be executed but it is not actively run

• running—operation is executing its logic

• completed—operation finished successfully

• failed—operation finished with an error.

Progress of an operation might be checked via progress field—it varies from 0.0 to 1.0. If job has completed at 80%, the progress would be 0.8. It is up to each operation to update its progress.

If an operation fails, whatever exceptions are thrown, they are captured in errors field. errors is an array; there might be 5 nodes and each might fail in its own way, so for each failure there would be a separate entry in errors array having different hostname—distinguishing the source of that error from each other.

snapshotTag is updated as well—there is automatically appended host id as well as timestamp—timestamp is equal for each node.

After a cluster is backed up, we may want to restore it. The restoration logic is similar to the backup one when it comes to a coordination/orchestration. If you want to restore a keyspace on a running Cassandra cluster, you just send one JSON to whatever node with this payload:

Again, globalRequest is true, storageLocation will be updated on node id, test-cluster and dc1 needs to be there. Other fields are self-explanatory—you can consult Swagger spec to learn more about them.

You may similarly GET operations to see there are multiple individual operations for each respective restoration phase.

There are restoration phases; these phases are passed per node, and each phase is initiated from coordinator only in case all nodes have passed that particular phase successfully.

• DOWNLOAD—download SSTables from remote location locally

• IMPORT—hardlinking (for Cassandra 3/4) or importing (for Cassandra 4 only) of SSTables from local destination of the respective node

• CLEANUP—optionally (by default turned on) delete downloaded SSTables

SSTables are truncated only after DOWNLOAD phase is over for all nodes.

import field specifies where downloaded SSTables will be located before they will be hardlinked or imported.

Each global restoration procedure has to start with restorationPhase: download. All other phases are handled automatically. This is the advantage of Icarus. In the case of Esop, one would have to execute each phase individually. In Icarus' case, a coordinator will take care of that transparently so in order to do a cluster-wide backup and restore on a running cluster, one just needs to ever send one JSON to whichever node and say it is a global request.

You may consult (and try) all operations via our Swagger spec.

To implement an operation against a Cassandra node, you need to, in general, do the following steps. We will guide a reader through cleanup operation which is very easy to grasp. In this simple example, an operation consists of three classes:

• CleanupOperation—the implementation of cleanup against Cassandra

• CleanupOperationRequest—this class represents your operation request sent to POST /operations. Feel free to model your operation request as you please, but type field has to be there. Each operation request is validated via Hibernate Validator so you may use javax.validation annotations on fields.

• Finally, we need a Guice module which is installed upon Icarus start:

Please note:

• this class extends Guice’s AbstractModule

• installOperationBindings static method is from com.instaclustr.operations.OperationBindings

• cleanup is operation type. You have to specify type field in your JSON request. Each operation is uniquely known and refered to through this type.

• Each operation has its implementation and its request—specified as other arguments to installation method.

After we have our operation module, we need to install it. This happens via com.instaclustr.icarus.Icarus#operationModules method.