Logging

​

Overview

Tilebox workflows are designed for distributed execution, making it essential to set up logging to a centralized system. Tilebox supports OpenTelemetry logging, which simplifies sending log messages from your tasks to a chosen backend. Collecting and visualizing logs from a distributed cluster of task runners in a tool like Axiom can look like this:

​

Configure logging

The Tilebox workflow SDKs include support for exporting OpenTelemetry logs. To enable logging, call the appropriate configuration functions during the startup of yourtask runner. Then, use the provided logger to send log messages from your tasks.

• Axiom
• OpenTelemetry-compatible backend
• Console

from tilebox.workflows import Client, Task, ExecutionContext
from tilebox.workflows.observability.logging import configure_otel_logging_axiom

# your own workflow:
from my_workflow import MyTask

def main():
    configure_otel_logging_axiom(
        # specify an Axiom dataset to export logs to
        dataset="my-axiom-logs-dataset",
        # along with an Axiom API key with ingest permissions for that dataset
        api_key="my-axiom-api-key",
    )

    # the task runner will export logs from 
    # the executed tasks to the specified dataset
    client = Client()
    runner = client.runner("dev-cluster", tasks=[MyTask])
    runner.run_forever()

if __name__ == "__main__":
    main()

​

Emitting log messages

Use the logger provided by the Tilebox SDK to emit log messages from your tasks. You can then use it to send log messages to the configured logging backend. Log messages emitted within a task’s execute method are also automatically recorded as span events for the current job trace.

import logging
from tilebox.workflows import Task, ExecutionContext
from tilebox.workflows.observability.logging import get_logger

logger = get_logger()

class MyTask(Task):
    def execute(self, context: ExecutionContext) -> None:
        # emit a log message to the configured OpenTelemetry backend
        logger.info("Hello world from configured logger!")

​

Logging task runner internals

Tilebox task runners also internally use a logger. By default, it’s set to the WARNING level, but you can change it by explicitly configuring a logger for the workflows client when constructing the task runner.

from tilebox.workflows import Client

from tilebox.workflows.observability.logging import configure_otel_logging_axiom
from tilebox.workflows.observability.logging import get_logger

# configure Axiom or another logging backend
configure_otel_logging_axiom(
    dataset="my-axiom-logs-dataset",
    api_key="my-axiom-api-key",
)

# configure a logger for the Tilebox client at the INFO level
client = Client()
client.configure_logger(get_logger(level=logging.INFO))

# now the task runner inherits this logger and uses
# it to emit its own internal log messages as well
runner = client.runner("dev-cluster", tasks=[MyTask])
runner.run_forever()