Observability
Effective observability is essential for building reliable workflows. Understanding and monitoring the execution of workflows and their tasks helps ensure correctness and efficiency. This section describes methods to gain insights into your workflow’s execution.
OpenTelemetry
Tilebox Workflows is designed with OpenTelemetry in mind, which provides a set of APIs and libraries for instrumenting, generating, collecting, and exporting telemetry data (metrics, logs, and traces) in distributed systems.
Tilebox Workflows currently supports OpenTelemetry for tracing and logging, with plans to include metrics in the future.
Integrations
Tilebox exports telemetry data using the OpenTelemetry Protocol. This allows you to send telemetry data to any OpenTelemetry-compatible backend, such as Axiom or Jaeger.
Axiom
Tilebox Workflows has built-in support for Axiom, a cloud-based observability and telemetry platform. The examples and screenshots in this section come from this integration.
To get started, sign up for a free Axiom account, create an axiom dataset for traces and logs, and generate an API key with ingest permissions for those datasets.
You can then configure Tilebox to export traces and logs to Axiom using configure_otel_tracing_axiom and configure_otel_logging_axiom.
Jaeger
Another popular option is Jaeger, a popular distributed tracing system. You can use the OpenTelemetry Collector to collect telemetry data from Tilebox Workflows and export it to Jaeger.
An all-in-one Jaeger environment can be spun up using Docker:
docker run --rm --name jaeger \
-p 5778:5778 \
-p 16686:16686 \
-p 4318:4318 \
jaegertracing/jaeger:2.9.0
configure_otel_tracing.
from tilebox.workflows import Client
from tilebox.workflows.observability.tracing import configure_otel_tracing
# your own workflow:
from my_workflow import MyTask
def main():
configure_otel_tracing(
# export traces to the local Jaeger instance
endpoint="http://localhost:4318",
)
# the following task runner generates traces for executed tasks and
# exports trace and span data to the specified endpoint
client = Client()
runner = client.runner(tasks=[MyTask])
runner.run_forever()
if __name__ == "__main__":
main()
