Clusters

​

Use Cases

You might want to organize task runners into clusters for many reasons, such as:

• Targeting a specific group of task runners for a particular job
• Reserving a group of task runners for specific purpose, such as only running a particular kind of batch processing jobs
• Set up different clusters for different environments (for example development, production)

Even Task runners belonging to the same cluster may have different capabilities, for example different tasks registered that they can run. Distributing tasks to different runners can also be achieved using only a single cluster, see the Distributed Execution example for more details.

If you have different task runners that all have the same tasks registered, you then can use clusters to only target specific task runners for a particular job.

​

Adding Task Runners to a Cluster

Task runners can be added to a cluster by specifying the clusters’ slug when registering a task runner.

​

Managing Clusters

Before registering a task runner, or submitting a job, you need to create a cluster. You can also list, fetch, and delete clusters as needed. The following sections illustrates how to do this.

To manage clusters, you first need to instantiate a cluster client. You can do this by calling the clusters method on the workflows’ client.

​

Creating a Cluster

To create a cluster, call the create method on the cluster client. You’ll need to provide a name for the cluster.

Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing')

​

Cluster Slug

Each cluster is identified by a unique name, consisting of the cluster’s name and a unique identifier. This identifier is generated automatically when the cluster is created. You can use this slug to reference the cluster in other operations, such as submitting a job or subtasks.

​

Listing Clusters

To list all the clusters that are available to you, you can use all:

[Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing'),
Cluster(slug='production-EifhUozDpwAJDL', display_name='Production')]

​

Fetching a specific Cluster

To fetch a specific cluster, you can use the fetch method and pass in a cluster’s slug:

Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing')

​

Deleting a Cluster

To delete a cluster, you can use the delete method and pass in a cluster’s slug:

​

Jobs spanning different clusters

When submitting a job you need to specify the cluster that the job should be submitted to. This allows you to target a specific group of task runners for a particular job. All sub-tasks within the job are submitted to the same cluster by default. This can also be manually overridden to submit sub-tasks to other clusters if needed. Check out the example below to see how this can be done.

This workflow requires at least two task runners to complete. One task runner must be in the “testing” cluster and the other must be in the “other-cluster” cluster. If for example no task runners are available in the “other-cluster” cluster, the task submitted to that cluster remains in the queue until a task runner becomes available. It won’t be executed by a task runner in the “testing” cluster, even though the task runner would have the capability to execute it, since it has the DummyTask registered.