What is a Task Runner?
Connecting to the Tilebox Workflows API
Selecting a cluster (optional)
Registering tasks
runner
method.Listening for new tasks
run_forever
method of the task runner to listen for new tasks until the task runner process is shut down.DownloadData
task, while the second picks up the ProcessData
task.
The DistributedWorkflow
does not require specific hardware, so it can be registered with both runners and executed by either one.
download_task_runner.py
and gpu_task_runner.py
are started, in parallel, on different machines with the required hardware for each. When DistributedWorkflow
is submitted, it executes on one of the two runners, and it’s submitted sub-tasks are handled by the appropriate runner.
In this case, since ProcessData
depends on DownloadData
, the GPU task runner remains idle until the download completion, then picks up the processing task.
context.submit_subtask
and then fails and is retried, those submitted sub-tasks of an earlier failed execution are automatically removed, ensuring that they can be safely submitted again when the task is retried.