# Agent Skills
Source: https://docs.tilebox.com/agents-and-ai-tools/agent-skills

Install Tilebox skills that teach AI agents how to work with Tilebox tools and workflows.

Tilebox skills are task-level instructions for AI agents. They explain how to use the Tilebox CLI, which command patterns to prefer, and how to approach common workflows such as dataset management and job monitoring.

## Install Tilebox skills and the CLI

Many skill operations require the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli),
which is why it's recommended to use our installation wizard that sets up both Skills and the CLI on your system.

```bash theme={"system"}
curl -fsSL https://install.tilebox.com/wizard.sh | sh
```

If you want to customize the skill installation directory, or select just a subset of the skills to install, you can use

```bash theme={"system"}
npx skills add tilebox/skills
```

<Columns>
  <Card title="Tilebox skills on GitHub" icon="github" href="https://github.com/tilebox/skills">
    Browse the source repository for the Tilebox agent skills.
  </Card>
</Columns>

After installation, your agent can load the relevant skill before working on a Tilebox task.

## Included skills

Tilebox skills cover common agent workflows around the CLI, datasets, workflows, jobs, and automations.

| Skill                              | What it covers                                                                                                                                                  |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `using-tilebox-cli`                | Authentication, `--json`, `jq`, `agent-context`, and documentation search                                                                                       |
| `managing-tilebox-datasets`        | Creating schemas, updating metadata, and querying datasets                                                                                                      |
| `managing-tilebox-jobs`            | Submitting jobs, monitoring status, reading logs, and inspecting spans                                                                                          |
| `working-with-tilebox-automations` | Working with triggers, automations, and storage locations                                                                                                       |
| `writing-tilebox-workflows`        | Writing workflow task classes, task graphs, runner definitions, caches, logs, and spans                                                                         |
| `releasing-tilebox-workflows`      | Initializing workflow projects, configuring `tilebox.workflow.toml`, building releases, publishing releases, deploying to clusters, and running release runners |

## How to use skills with agents

Ask your agent to load the most specific Tilebox skill for the task. For example, use `managing-tilebox-datasets` for schema work and `managing-tilebox-jobs` for workflow execution or observability tasks.

<AgentPrompt>
  Load the relevant Tilebox skills. Use the Tilebox CLI to submit a Sentinel-2 mosaic job, then monitor it until it completes.
</AgentPrompt>

Skills work best with the Tilebox CLI. The CLI gives the agent repeatable terminal commands, and skills tell it how to combine those commands safely. Use MCP as an alternative when the agent runs in a web or chat environment without practical terminal access.

For Python workflow release work, ask the agent to use both `writing-tilebox-workflows` and `releasing-tilebox-workflows`. For a new project, the typical loop starts with `tilebox workflow init`. After that, the agent edits tasks, builds a release, publishes it, deploys it to a development cluster, runs `tilebox runner start`, submits a test job, and inspects logs or spans before iterating.


# AI interfaces
Source: https://docs.tilebox.com/agents-and-ai-tools/ai-interfaces

Choose the right Tilebox interface for AI-assisted work.

Tilebox gives agents multiple ways to get context and take action. The right interface depends on whether the agent needs live Tilebox access, repeatable terminal commands, workflow instructions, or documentation-only context.

## Interface guide

| Interface                                                 | Use it for                                                                                  | What it gives the agent                                               |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [Tilebox CLI](/agents-and-ai-tools/tilebox-cli)           | Default setup for coding agents, terminal operations, automation, and CI-friendly workflows | Structured command output, command discovery, and predictable inputs  |
| [Agent Skills](/agents-and-ai-tools/agent-skills)         | Repeated Tilebox tasks across datasets, jobs, and automations                               | Instructions for composing CLI commands into end-to-end workflows     |
| [Tilebox MCP](/agents-and-ai-tools/tilebox-mcp)           | Web-based agents, chat tools, or MCP-capable clients where terminal access is limited       | Authenticated tools for datasets, workflows, and documentation search |
| [Tilebox SDKs](/sdks/introduction)                        | Application code and production integrations                                                | Language-native APIs for Python and Go                                |
| [`llms-full.txt`](https://docs.tilebox.com/llms-full.txt) | Documentation-only context                                                                  | A static markdown bundle of the Tilebox documentation                 |

## How the interfaces fit together

Use the CLI when an agent can run terminal commands. This is the default interface for repository-based coding agents because commands are explicit, repeatable, and easy to inspect. Use skills when the agent needs guidance for multi-step Tilebox tasks, such as creating datasets or monitoring workflow jobs.

Use the MCP server when an agent cannot easily run terminal commands. This is common for web-based agents and chat tools where the agent can call hosted tools but cannot access a shell. MCP is also useful when you want an AI client to access Tilebox documentation search and live Tilebox tools without installing local software.

The interfaces are complementary. For example, a coding agent can use the CLI to inspect available commands with `tilebox agent-context` and use the `managing-tilebox-jobs` skill to monitor a submitted workflow job. A web-based agent can use the MCP server for documentation search and live Tilebox access instead.

## Recommended defaults

For AI-assisted development in a codebase, install the CLI and Tilebox skills. Add MCP when your agent runs outside a terminal environment or when you prefer a hosted tool connection from a web or chat interface.


# Agents and AI tools
Source: https://docs.tilebox.com/agents-and-ai-tools/overview

Learn how Tilebox gives coding agents access to the same data, workflow, job, and observability tools that developers use.

Tilebox gives agents the same operational context that developers use: current documentation, authenticated access to Tilebox resources, and predictable tools for running actions. This section explains how to connect agents to Tilebox, which interface to use for each task, and how the Tilebox CLI and skills help agents work with datasets and workflows.

## How Tilebox supports agents

Tilebox supports agents through a CLI-based setup. The CLI gives agents a deterministic terminal interface with structured output, machine-readable command discovery, and file-based inputs. Agent skills teach agents how to compose CLI commands into common Tilebox tasks, while the MCP server is available for AI tools where terminal access is limited, such as web-based chat agents.

<Columns>
  <Card title="Onboard your agent" icon="rocket" href="/onboard-your-agent">
    Configure an agent with the Tilebox CLI, Tilebox skills, and documentation context.
  </Card>

  <Card title="AI interfaces" icon="route" href="/agents-and-ai-tools/ai-interfaces">
    Decide when to use MCP, the CLI, skills, SDKs, or documentation context.
  </Card>

  <Card title="Tilebox CLI" icon="terminal" href="/agents-and-ai-tools/tilebox-cli">
    Install the CLI, authenticate it, and use agent-friendly command patterns.
  </Card>

  <Card title="Agent Skills" icon="wand-magic-sparkles" href="/agents-and-ai-tools/agent-skills">
    Install Tilebox skills that teach agents how to work with Tilebox tools.
  </Card>

  <Card title="Tilebox MCP" icon="plug" href="/agents-and-ai-tools/tilebox-mcp">
    Connect AI tools to the Tilebox MCP server when terminal access is not available.
  </Card>

  <Card title="Agentic workflow iteration" icon="robot" href="/guides/workflows/agentic-workflow-iteration">
    Use an agent to edit workflow code, publish releases, deploy them to a dev cluster, and inspect jobs.
  </Card>
</Columns>

## Recommended setup

Use [Onboard your agent](/onboard-your-agent) if you want an AI coding agent to help with Tilebox work in a repository. It covers the default setup with CLI installation, CLI authentication, Tilebox skills, and documentation context. Use the MCP setup when your agent runs in a web or chat environment where commands and terminals are not easy to use.


# Tilebox CLI
Source: https://docs.tilebox.com/agents-and-ai-tools/tilebox-cli

Install, authenticate, and use the Tilebox CLI for developer workflows, operations, and AI-assisted development.

The Tilebox CLI is a command-line interface for developers, operators, agents, and automation. Use it to inspect Tilebox resources, manage datasets and workflows, submit jobs, start release runners, and run operational commands from the shell.

The CLI is also the default Tilebox interface for coding agents that can run terminal commands. It keeps actions visible in the shell, supports structured output, and pairs with Tilebox skills for multi-step tasks.

## Install the CLI

Install the Tilebox CLI with the install script.

```bash theme={"system"}
curl -fsSL https://install.tilebox.com/cli.sh | sh
```

<Tip>
  If you want to install the Tilebox CLI and Agent Skills together, use the setup wizard instead.

  ```bash theme={"system"}
  curl -fsSL https://install.tilebox.com/wizard.sh | sh
  ```
</Tip>

## Authenticate

The CLI reads the `TILEBOX_API_KEY` environment variable. Create an API key in the [Tilebox Console](https://console.tilebox.com/settings/api-keys), then export it in the shell or agent environment that runs CLI commands.

```bash theme={"system"}
export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY"
```

<Note>
  Alternatively, you can also provide the API key, or override the environment variable with the `--api-key` flag in each command.
</Note>

## Discover commands with agent-context

Agents should inspect the CLI instead of guessing command names and flags. The `agent-context` command returns machine-readable information about the available command tree, arguments, flags, and descriptions.

```bash theme={"system"}
tilebox agent-context
```

Use this before asking an agent to create datasets, submit jobs, or manage automations with the CLI.

## Prefer JSON output

Use `--json` when the agent needs to parse command output. Structured output is easier for agents to inspect, filter, and pass to follow-up commands than terminal tables.

```bash theme={"system"}
tilebox dataset list --json
```

## Scaffold workflow projects

Use the CLI to create a Tilebox workflow and scaffold a Python release project.

```bash theme={"system"}
tilebox workflow init --name "Scene QA" --json
```

The command creates the remote workflow, writes `tilebox.workflow.toml`, creates Python project files, adds the `tilebox` dependency, and runs `uv sync`. See [Project Structure](/workflows/build-and-deploy/project-structure) for the generated files and release project layout.

## Use files and standard input for generated input

For larger inputs, prefer file-based flags instead of long shell-quoted strings. The CLI supports input patterns such as `--schema-file`, `--input-file`, `--spatial-extent-file`, and `--description-file`. Many file flags also support `-` for reading from standard input.

```bash theme={"system"}
tilebox dataset create --schema-file schema.json
```

```bash theme={"system"}
# Read from stdin.
cat schema.json | tilebox dataset create --schema-file -
```

This pattern is safer for agents because generated JSON, geometry, and descriptions do not need to be embedded in one fragile shell command.

## Use the CLI with Tilebox skills

The Tilebox skills include guidance for using the CLI in agent workflows. Install them when you want an agent to combine CLI commands into higher-level tasks such as managing datasets, monitoring jobs, or configuring automations.

```bash theme={"system"}
curl -fsSL https://install.tilebox.com/skills.sh | sh
```

After installing the skills, ask your agent to inspect `tilebox agent-context` and follow the relevant Tilebox skill before running commands that change resources. If your agent cannot use a terminal, configure the [Tilebox MCP server](/agents-and-ai-tools/tilebox-mcp) instead.


# Tilebox MCP
Source: https://docs.tilebox.com/agents-and-ai-tools/tilebox-mcp

Configure the Tilebox MCP server for AI tools that support the Model Context Protocol.

The Tilebox MCP server connects AI tools to Tilebox through the Model Context Protocol. It exposes tools for working with Tilebox datasets and workflows, and it includes documentation search so agents can retrieve current Tilebox context before answering questions or taking action.

Use MCP when your agent cannot easily use terminal commands. For coding agents with shell access, the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) and [Agent Skills](/agents-and-ai-tools/agent-skills) are the default setup. MCP is a good fit for web-based agents, chat tools, or hosted AI clients that support MCP but do not run local commands.

## What the MCP server provides

The MCP server at `https://mcp.tilebox.com/mcp` provides tools for:

* Accessing and interacting with Tilebox datasets and workflows
* Searching the Tilebox documentation

When an MCP-capable client has the server configured, it can invoke relevant tools during a conversation. For example, when you ask about a dataset, the agent can query the Tilebox MCP server for the current schema and include that information in its response.

## Authentication

The Tilebox MCP server uses OAuth. When your AI client connects, you are redirected to sign in with your Tilebox account and select a team. You do not need to configure API keys or custom headers for MCP authentication.

## Configure an MCP client

Add the Tilebox MCP server using the HTTP transport when your AI client supports MCP.

```json theme={"system"}
{
  "mcpServers": {
    "tilebox": {
      "url": "https://mcp.tilebox.com/mcp"
    }
  }
}
```

## Configure Claude Code

Run the following command to add the Tilebox MCP server to Claude Code:

```bash theme={"system"}
claude mcp add --transport http "Tilebox" https://mcp.tilebox.com/mcp
```

## Use multiple teams

If you need to access multiple Tilebox teams, add a separate MCP server entry for each team. Each entry goes through its own OAuth flow and can be authorized for a different team.

```json theme={"system"}
{
  "mcpServers": {
    "tilebox-team1": {
      "url": "https://mcp.tilebox.com/mcp"
    },
    "tilebox-team2": {
      "url": "https://mcp.tilebox.com/mcp"
    }
  }
}
```

<Tip>
  Use clear server names when configuring multiple teams so your agent can choose the correct Tilebox workspace for each task.
</Tip>


# As
Source: https://docs.tilebox.com/api-reference/go/datasets/As



```go theme={"system"}
func As[T proto.Message](seq iter.Seq2[[]byte, error]) iter.Seq2[T, error]
```

Convert a sequence of bytes into a sequence of `proto.Message`.

Useful to convert the output of [`Datapoints.Query`](/api-reference/go/datasets/Datapoints.Query) into a sequence of `proto.Message`.

## Parameters

<ParamField type="iter.Seq2[[]byte, error]">
  The sequence of bytes to convert
</ParamField>

## Returns

A sequence of `proto.Message` or an error if any.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    datasets "github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/query"
  )

  startDate := time.Date(2014, 10, 4, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2021, 2, 24, 0, 0, 0, 0, time.UTC) 
  queryInterval := query.NewTimeInterval(startDate, endDate)

  seq := datasets.As[*v1.Sentinel1Sar](
      client.Datapoints.Query(
        ctx,
        datasetID,
        datasets.WithCollectionIDs(collectionID),
        datasets.WithTemporalExtent(queryInterval),
      ),
  )
  ```
</RequestExample>


# Collect
Source: https://docs.tilebox.com/api-reference/go/datasets/Collect



```go theme={"system"}
func Collect[K any](seq iter.Seq2[K, error]) ([]K, error)
```

Convert any sequence into a slice.

It returns an error if any element in the sequence has a non-nil error.

## Parameters

<ParamField type="iter.Seq2[K, error]">
  The sequence to convert.
</ParamField>

## Returns

A slice of `K` or an error if any.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    datasets "github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/query"
  )

  startDate := time.Date(2014, 10, 4, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2021, 2, 24, 0, 0, 0, 0, time.UTC) 
  queryInterval := query.NewTimeInterval(startDate, endDate)

  datapoints, err := datasets.Collect(datasets.As[*v1.Sentinel1Sar](
      client.Datapoints.Query(
        ctx,
        datasetID,
        datasets.WithCollectionIDs(collectionID),
        datasets.WithTemporalExtent(queryInterval),
      ),
  ))
  ```
</RequestExample>


# CollectAs
Source: https://docs.tilebox.com/api-reference/go/datasets/CollectAs



```go theme={"system"}
func CollectAs[T proto.Message](seq iter.Seq2[[]byte, error]) ([]T, error)
```

Convert a sequence of bytes into a slice of `proto.Message`.

Useful to convert the output of [`Datapoints.Query`](/api-reference/go/datasets/Datapoints.Query) into a slice of `proto.Message`.

This a convenience function for `Collect(As[T](seq))`.

## Parameters

<ParamField type="iter.Seq2[[]byte, error]">
  The sequence of bytes to convert
</ParamField>

## Returns

A slice of `proto.Message` or an error if any.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    datasets "github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/query"
  )

  startDate := time.Date(2014, 10, 4, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2021, 2, 24, 0, 0, 0, 0, time.UTC) 
  queryInterval := query.NewTimeInterval(startDate, endDate)

  datapoints, err := datasets.CollectAs[*v1.Sentinel1Sar](
      client.Datapoints.Query(
        ctx,
        datasetID,
        datasets.WithCollectionIDs(collectionID),
        datasets.WithTemporalExtent(queryInterval),
      ),
  )
  ```
</RequestExample>


# Collections.Create
Source: https://docs.tilebox.com/api-reference/go/datasets/Collections.Create



```go theme={"system"}
func (collectionClient) Create(
    ctx context.Context,
    datasetID uuid.UUID,
    collectionName string,
) (*datasets.Collection, error)
```

Create a collection in the dataset.

## Parameters

<ParamField type="uuid.UUID">
  The id of the dataset
</ParamField>

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

The created collection object.

<RequestExample>
  ```go Go theme={"system"}
  collection, err := client.Collections.Create(ctx,
      datasetID,
      "My-collection",
  )
  ```
</RequestExample>


# Collections.Delete
Source: https://docs.tilebox.com/api-reference/go/datasets/Collections.Delete



```go theme={"system"}
func (collectionClient) Delete(
    ctx context.Context,
    datasetID uuid.UUID,
    collectionID uuid.UUID,
) error
```

Delete a collection by its id.

## Parameters

<ParamField type="uuid.UUID">
  The id of the dataset
</ParamField>

<ParamField type="uuid.UUID">
  The id of the collection
</ParamField>

## Returns

An error if the collection could not be deleted.

<RequestExample>
  ```go Go theme={"system"}
  err := client.Collections.Delete(ctx,
      datasetID,
      collectionID,
  )
  ```
</RequestExample>

## Errors

<ParamField type="No such dataset">
  The specified dataset does not exist.
</ParamField>

<ParamField type="No such collection">
  The specified collection does not exist.
</ParamField>


# Collections.Get
Source: https://docs.tilebox.com/api-reference/go/datasets/Collections.Get



```go theme={"system"}
func (collectionClient) Get(
    ctx context.Context,
    datasetID uuid.UUID,
    name string,
) (*datasets.Collection, error)
```

Get a collection by name from a dataset.

## Parameters

<ParamField type="uuid.UUID">
  The id of the dataset
</ParamField>

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

A collection object.

<RequestExample>
  ```go Go theme={"system"}
  collection, err := client.Collections.Get(ctx,
      datasetID,
      "My-collection",
  )
  ```
</RequestExample>

## Errors

<ParamField type="No such dataset">
  The specified dataset does not exist.
</ParamField>


# Collections.GetOrCreate
Source: https://docs.tilebox.com/api-reference/go/datasets/Collections.GetOrCreate



```go theme={"system"}
func (collectionClient) GetOrCreate(
    ctx context.Context,
    datasetID uuid.UUID,
    name string,
) (*datasets.Collection, error)
```

Get or create a collection by its name. If the collection does not exist, it will be created.

## Parameters

<ParamField type="uuid.UUID">
  The id of the dataset
</ParamField>

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

A collection object.

<RequestExample>
  ```go Go theme={"system"}
  collection, err := client.Collections.GetOrCreate(ctx,
      datasetID,
      "My-collection",
  )
  ```
</RequestExample>


# Collections.List
Source: https://docs.tilebox.com/api-reference/go/datasets/Collections.List



```go theme={"system"}
func (collectionClient) List(
    ctx context.Context,
    datasetID uuid.UUID,
) ([]*datasets.Collection, error)
```

List the available collections in a dataset.

## Parameters

<ParamField type="uuid.UUID">
  The id of the dataset
</ParamField>

## Returns

A list of collection objects.

<RequestExample>
  ```go Go theme={"system"}
  collections, err := client.Collections.List(ctx,
      datasetID,
  )
  ```
</RequestExample>

## Errors

<ParamField type="No such dataset">
  The specified dataset does not exist.
</ParamField>


# Datasets.Create
Source: https://docs.tilebox.com/api-reference/go/datasets/Create



```go theme={"system"}
func (datasetClient) Create(
    ctx context.Context,
    kind datasets.DatasetKind,
    codeName string,
    name string,
    fields []datasets.Field,
    options ...datasets.DatasetOption,
) (*datasets.Dataset, error)
```

Create a dataset with the given code name, display name, schema kind, and custom fields.

## Parameters

<ParamField type="datasets.DatasetKind">
  The dataset kind.
</ParamField>

<ParamField type="string">
  The stable code identifier for the dataset.
</ParamField>

<ParamField type="string">
  The display name of the dataset.
</ParamField>

<ParamField type="[]datasets.Field">
  The custom fields in the dataset schema.
</ParamField>

<ParamField type="[]datasets.DatasetOption">
  Options for dataset metadata.
</ParamField>

## Options

<ParamField>
  Set a short dataset summary.
</ParamField>

<ParamField>
  Set the dataset's markdown description.
</ParamField>

## Dataset kinds

<ParamField type="datasets.DatasetKind">
  A dataset that contains timestamp, ID, and ingestion time fields.
</ParamField>

<ParamField type="datasets.DatasetKind">
  A dataset that contains timestamp, ID, ingestion time, and geometry fields.
</ParamField>

## Returns

The created dataset object.

<RequestExample>
  ```go Go theme={"system"}
  dataset, err := client.Datasets.Create(ctx,
      datasets.KindSpatiotemporal,
      "my_catalog",
      "My catalog",
      []datasets.Field{
          field.String("source").Description("Source system"),
          field.Float64("cloud_cover"),
      },
      datasets.WithSummary("Scenes prepared for analysis"),
  )
  ```
</RequestExample>


# Datasets.CreateOrUpdate
Source: https://docs.tilebox.com/api-reference/go/datasets/CreateOrUpdate



```go theme={"system"}
func (datasetClient) CreateOrUpdate(
    ctx context.Context,
    kind datasets.DatasetKind,
    codeName string,
    name string,
    fields []datasets.Field,
    options ...datasets.DatasetOption,
) (*datasets.Dataset, error)
```

Create a dataset or update an existing dataset if a dataset with the given `codeName` already exists.

If the dataset already exists, Tilebox applies the same schema update rules as a direct update. New fields can be added to non-empty datasets. Breaking schema changes are only allowed for empty datasets.

## Parameters

<ParamField type="datasets.DatasetKind">
  The dataset kind.
</ParamField>

<ParamField type="string">
  The stable code identifier for the dataset.
</ParamField>

<ParamField type="string">
  The display name of the dataset.
</ParamField>

<ParamField type="[]datasets.Field">
  The custom fields in the dataset schema.
</ParamField>

<ParamField type="[]datasets.DatasetOption">
  Options for dataset metadata.
</ParamField>

## Options

<ParamField>
  Set a short dataset summary.
</ParamField>

<ParamField>
  Set the dataset's markdown description.
</ParamField>

## Dataset kinds

<ParamField type="datasets.DatasetKind">
  A dataset that contains timestamp, ID, and ingestion time fields.
</ParamField>

<ParamField type="datasets.DatasetKind">
  A dataset that contains timestamp, ID, ingestion time, and geometry fields.
</ParamField>

## Field types

<ParamField type="Field">
  A string field
</ParamField>

<ParamField type="Field">
  A bytes field
</ParamField>

<ParamField type="Field">
  A boolean field
</ParamField>

<ParamField type="Field">
  A 64-bit signed integer field
</ParamField>

<ParamField type="Field">
  A 64-bit unsigned integer field
</ParamField>

<ParamField type="Field">
  A 64-bit floating-point number field
</ParamField>

<ParamField type="Field">
  A duration field
</ParamField>

<ParamField type="Field">
  A timestamp field
</ParamField>

<ParamField type="Field">
  A UUID field
</ParamField>

<ParamField type="Field">
  A geometry field
</ParamField>

## Field options

<ParamField>
  Indicate that the field is an array
</ParamField>

<ParamField>
  Set the description of the field to provide more context and details about the data
</ParamField>

<ParamField>
  Set the example value of the field for documentation purposes
</ParamField>

## Returns

The created or updated dataset object.

<RequestExample>
  ```go Go theme={"system"}
  dataset, err := client.Datasets.CreateOrUpdate(ctx,
      datasets.KindSpatiotemporal,
      "my_catalog",
      "My catalog",
      []datasets.Field{
          field.String("field1"),
          field.Int64("field2").Repeated(),
          field.Geometry("field3").Description("Field 3").ExampleValue("Value 3"),
      },
      datasets.WithSummary("Scenes prepared for analysis"),
  )
  ```
</RequestExample>


# DatapointDecoder.Unmarshal
Source: https://docs.tilebox.com/api-reference/go/datasets/DatapointDecoder.Unmarshal



```go theme={"system"}
func (d datasets.DatapointDecoder) Unmarshal(
    descriptor *datasets.DatapointDescriptor,
    data []byte,
) (map[string]any, error)
```

Decode a raw protobuf datapoint into a JSON-like map with configurable protobuf decoding options.

## Parameters

<ParamField type="*datasets.DatapointDescriptor">
  The descriptor returned by [`NewDatapointDescriptor`](/api-reference/go/datasets/NewDatapointDescriptor).
</ParamField>

<ParamField type="[]byte">
  The raw protobuf datapoint bytes returned by a datapoint query.
</ParamField>

## Decoder options

<ParamField type="bool">
  Allow messages that are missing required fields.
</ParamField>

<ParamField type="bool">
  Ignore unknown fields in the raw protobuf message.
</ParamField>

<ParamField>
  Override the resolver used to look up message and extension types.
</ParamField>

<ParamField type="int">
  Limit how deeply nested messages may be decoded. A zero value uses the protobuf default.
</ParamField>

## Returns

A map of datapoint fields, or an error if the raw datapoint cannot be decoded.

<RequestExample>
  ```go Go theme={"system"}
  decoder := datasets.DatapointDecoder{
      DiscardUnknown: true,
  }

  datapoint, err := decoder.Unmarshal(descriptor, data)
  if err != nil {
      return err
  }
  ```
</RequestExample>


# Datapoints.Delete
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.Delete



```go theme={"system"}
func (datapointClient) Delete(
    ctx context.Context,
    collectionID uuid.UUID,
    datapoints any,
) (int64, error)
```

Delete data points from a collection.

Data points are identified and deleted by their ids.

## Parameters

<ParamField type="uuid.UUID">
  The id of the collection
</ParamField>

<ParamField type="[]proto.Message">
  The datapoints to delete from the collection
</ParamField>

## Returns

The number of data points that were deleted.

<RequestExample>
  ```go Go theme={"system"}
  var datapoints []*v1.Sentinel1Sar
  // assuming the slice is filled with datapoints

  numDeleted, err := client.Datapoints.Delete(ctx,
      collectionID,
      datapoints,
  )
  ```
</RequestExample>


# Datapoints.DeleteIDs
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.DeleteIDs



```go theme={"system"}
func (datapointClient) DeleteIDs(
    ctx context.Context,
    collectionID uuid.UUID,
    datapointIDs []uuid.UUID,
) (int64, error)
```

Delete data points from a collection.

## Parameters

<ParamField type="uuid.UUID">
  The id of the collection
</ParamField>

<ParamField type="[]uuid.UUID">
  The ids of the data points to delete from the collection
</ParamField>

## Returns

The number of data points that were deleted.

<RequestExample>
  ```go Go theme={"system"}
  numDeleted, err := client.Datapoints.DeleteIDs(ctx,
      collectionID,
      []uuid.UUID{
        uuid.MustParse("0195c87a-49f6-5ffa-e3cb-92215d057ea6"),
        uuid.MustParse("0195c87b-bd0e-3998-05cf-af6538f34957"),
      },
  )
  ```
</RequestExample>


# Datapoints.GetInto
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.GetInto



```go theme={"system"}
func (datapointClient) GetInto(
  ctx context.Context,
  datasetID uuid.UUID,
  datapointID uuid.UUID,
  datapoint proto.Message,
  options ...QueryOption,
) error
```

Get a datapoint by its ID from one or more collections of the same dataset.

The data point is stored in the `datapoint` parameter.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the dataset to query.
</ParamField>

<ParamField type="uuid.UUID">
  The id of the datapoint to query
</ParamField>

<ParamField type="proto.Message">
  The datapoint to query into
</ParamField>

<ParamField type="[]QueryOption">
  Options for querying data points.
</ParamField>

## Options

<ParamField>
  Restrict the lookup to specific dataset collections by collection object.
</ParamField>

<ParamField>
  Restrict the lookup to specific dataset collections by collection ID.
</ParamField>

<ParamField>
  Skip the data when querying datapoint.
  If set, only the required and auto-generated fields will be returned.
</ParamField>

## Returns

An error if data point could not be queried.

<RequestExample>
  ```go Go theme={"system"}
  var datapoint v1.Sentinel1Sar
  err = client.Datapoints.GetInto(ctx,
    dataset.ID,
    datapointID,
    &datapoint,
    datasets.WithCollectionIDs(collection.ID),
  )
  ```
</RequestExample>


# Datapoints.Ingest
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.Ingest



```go theme={"system"}
func (datapointClient) Ingest(
    ctx context.Context,
    collectionID uuid.UUID,
    datapoints any,
    allowExisting bool,
) (*datasets.IngestResponse, error)
```

Ingest data points into a collection.

## Parameters

<ParamField type="uuid.UUID">
  The id of the collection
</ParamField>

<ParamField type="*[]proto.Message">
  The datapoints to ingest
</ParamField>

<ParamField type="bool">
  Datapoint fields are used to generate a deterministic unique `UUID` for each
  datapoint in a collection. Duplicate data points result in the same ID being generated.
  If `allowExisting` is `true`, `ingest` will skip those datapoints, since they already exist.
  If `allowExisting` is `false`, `ingest` will raise an error if any of the generated datapoint IDs already exist.
</ParamField>

## Returns

An `IngestResponse` with `NumCreated`, `NumExisting`, and `DatapointIDs`. `DatapointIDs` includes the IDs of ingested datapoints and, when `allowExisting` is `true`, the IDs of datapoints that already existed.

<RequestExample>
  ```go Go theme={"system"}
  datapoints := []*v1.Modis{
    v1.Modis_builder{
      Time:        timestamppb.New(time.Now()),
      GranuleName: proto.String("Granule 1"),
    }.Build(),
    v1.Modis_builder{
      Time:        timestamppb.New(time.Now().Add(-5 * time.Hour)),
      GranuleName: proto.String("Past Granule 2"),
    }.Build(),
  }

  ingestResponse, err := client.Datapoints.Ingest(ctx,
      collectionID,
      &datapoints,
      false,
  )
  ```
</RequestExample>

## Errors

<ParamField type="found existing datapoints with same id">
  If `allowExisting` is `false` and any datapoints already exist.
</ParamField>


# Datapoints.Query
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.Query



```go theme={"system"}
func (datapointClient) Query(
    ctx context.Context,
    datasetID uuid.UUID,
    options ...datasets.QueryOption,
) iter.Seq2[[]byte, error]
```

Query datapoints from one or more collections of the same dataset.

The datapoints are lazily queried across pages and returned as a sequence of bytes.
The output sequence can be transformed into a typed `proto.Message` using [CollectAs](/api-reference/go/datasets/CollectAs) or [As](/api-reference/go/datasets/As) functions.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the dataset to query.
</ParamField>

<ParamField type="[]datasets.QueryOption">
  Options for querying datapoints.
</ParamField>

## Options

<ParamField>
  Specify the time interval for which data should be queried.
  Right now, a temporal extent is required for every query.
</ParamField>

<ParamField>
  Specify the geographical extent in which to query data.
  Optional, if not specified the query will return all results found globally.
</ParamField>

<ParamField>
  Specify a geographical extent with an explicit spatial filter mode and coordinate system.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection object.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection ID.
</ParamField>

<ParamField>
  Skip the data when querying datapoints.
  If set, only the required and auto-generated fields will be returned.
</ParamField>

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the total number of datapoints yielded by the sequence.
</ParamField>

## Returns

A sequence of bytes containing the requested data points as bytes.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    datasets "github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/query"
  )

  startDate := time.Date(2014, 10, 4, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2021, 2, 24, 0, 0, 0, 0, time.UTC) 
  queryInterval := query.NewTimeInterval(startDate, endDate)

  datapoints, err := datasets.CollectAs[*v1.Sentinel1Sar](
      client.Datapoints.Query(
        ctx,
        dataset.ID,
        datasets.WithCollectionIDs(collection.ID),
        datasets.WithTemporalExtent(queryInterval),
      ),
  )
  ```
</RequestExample>


# Datapoints.QueryInto
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.QueryInto



```go theme={"system"}
func (datapointClient) QueryInto(
    ctx context.Context,
    datasetID uuid.UUID,
    datapoints any,
    options ...datasets.QueryOption,
) error
```

Query datapoints from one or more collections of the same dataset into a slice.

QueryInto is a convenience function for [Query](/api-reference/go/datasets/Datapoints.Query), when no manual pagination or custom iteration is required.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the dataset to query.
</ParamField>

<ParamField type="*[]proto.Message">
  The datapoints to query into
</ParamField>

<ParamField type="[]datasets.QueryOption">
  Options for querying data points
</ParamField>

## Options

<ParamField>
  Specify the time interval for which data should be queried.
  Right now, a temporal extent is required for every query.
</ParamField>

<ParamField>
  Specify the geographical extent in which to query data.
  Optional, if not specified the query will return all results found globally.
</ParamField>

<ParamField>
  Specify a geographical extent with an explicit spatial filter mode and coordinate system.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection object.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection ID.
</ParamField>

<ParamField>
  Skip the data when querying datapoints.
  If set, only the required and auto-generated fields will be returned.
</ParamField>

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the total number of datapoints returned.
</ParamField>

## Returns

An error if data points could not be queried.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    datasets "github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/query"
  )

  startDate := time.Date(2014, 10, 4, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2021, 2, 24, 0, 0, 0, 0, time.UTC) 
  queryInterval := query.NewTimeInterval(startDate, endDate)

  var datapoints []*v1.Sentinel1Sar
  err := client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithCollectionIDs(collection.ID),
      datasets.WithTemporalExtent(queryInterval),
  )
  ```
</RequestExample>


# Datapoints.QueryPage
Source: https://docs.tilebox.com/api-reference/go/datasets/Datapoints.QueryPage



```go theme={"system"}
func (datapointClient) QueryPage(
    ctx context.Context,
    datasetID uuid.UUID,
    options ...datasets.QueryOption,
) (*datasets.DatapointPage, error)
```

Query a single page of datapoints from one or more collections of the same dataset.

Use `QueryPage` when you need manual pagination. Use [`Datapoints.Query`](/api-reference/go/datasets/Datapoints.Query) for automatic lazy pagination.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the dataset to query.
</ParamField>

<ParamField type="[]datasets.QueryOption">
  Options for querying datapoints.
</ParamField>

## Options

<ParamField>
  Specify the time or datapoint ID interval to query.
</ParamField>

<ParamField>
  Specify the geographical extent to query.
</ParamField>

<ParamField>
  Specify a geographical extent with an explicit spatial filter mode and coordinate system.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection object.
</ParamField>

<ParamField>
  Restrict the query to specific dataset collections by collection ID.
</ParamField>

<ParamField>
  Skip datapoint data and return only required and generated fields.
</ParamField>

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the number of datapoints returned in this page.
</ParamField>

## Returns

A `DatapointPage` with raw protobuf datapoints in `Datapoints` and an optional `NextCursor` for the next page.

<RequestExample>
  ```go Go theme={"system"}
  page, err := client.Datapoints.QueryPage(ctx,
      dataset.ID,
      datasets.WithTemporalExtent(queryInterval),
      datasets.WithCollectionIDs(collection.ID),
      datasets.WithLimit(100),
  )
  if err != nil {
      return err
  }

  if page.NextCursor != nil {
      nextPage, err := client.Datapoints.QueryPage(ctx,
          dataset.ID,
          datasets.WithTemporalExtent(queryInterval),
          datasets.WithCollectionIDs(collection.ID),
          datasets.WithCursor(page.NextCursor),
          datasets.WithLimit(100),
      )
      _ = nextPage
      _ = err
  }
  ```
</RequestExample>


# Datasets.Get
Source: https://docs.tilebox.com/api-reference/go/datasets/Get



```go theme={"system"}
func (datasetClient) Get(
    ctx context.Context,
    slug string,
) (*datasets.Dataset, error)
```

Get a dataset by its slug.

## Parameters

<ParamField type="string">
  The slug of the dataset
</ParamField>

## Returns

A dataset object.

<RequestExample>
  ```go Go theme={"system"}
  s1_sar, err := client.Datasets.Get(ctx,
      "open_data.copernicus.sentinel1_sar"
  )
  ```
</RequestExample>

## Errors

<ParamField type="No such dataset">
  The specified dataset does not exist.
</ParamField>


# Datasets.List
Source: https://docs.tilebox.com/api-reference/go/datasets/List



```go theme={"system"}
func (datasetClient) List(ctx context.Context) ([]*datasets.Dataset, error)
```

Fetch all available datasets.

## Returns

A list of all available datasets.

<RequestExample>
  ```go Go theme={"system"}
  datasets, err := client.Datasets.List(ctx)
  ```
</RequestExample>


# datasets.NewDatapointDescriptor
Source: https://docs.tilebox.com/api-reference/go/datasets/NewDatapointDescriptor



```go theme={"system"}
func NewDatapointDescriptor(dataset *datasets.Dataset) (*datasets.DatapointDescriptor, error)
```

Create a reusable descriptor for decoding raw protobuf datapoints from a loaded dataset.

Use this helper when you want to query datasets without generated Go protobuf types.

## Parameters

<ParamField type="*datasets.Dataset">
  A dataset returned by `client.Datasets.Get` or another dataset client method.
</ParamField>

## Returns

A `DatapointDescriptor` that can be passed to [`UnmarshalDatapoint`](/api-reference/go/datasets/UnmarshalDatapoint) or [`DatapointDecoder.Unmarshal`](/api-reference/go/datasets/DatapointDecoder.Unmarshal).

<RequestExample>
  ```go Go theme={"system"}
  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel1_sar")
  if err != nil {
      return err
  }

  descriptor, err := datasets.NewDatapointDescriptor(dataset)
  if err != nil {
      return err
  }
  ```
</RequestExample>


# datasets.UnmarshalDatapoint
Source: https://docs.tilebox.com/api-reference/go/datasets/UnmarshalDatapoint



```go theme={"system"}
func UnmarshalDatapoint(
    descriptor *datasets.DatapointDescriptor,
    data []byte,
) (map[string]any, error)
```

Decode a raw protobuf datapoint into a JSON-like map using a dataset descriptor.

## Parameters

<ParamField type="*datasets.DatapointDescriptor">
  The descriptor returned by [`NewDatapointDescriptor`](/api-reference/go/datasets/NewDatapointDescriptor).
</ParamField>

<ParamField type="[]byte">
  The raw protobuf datapoint bytes returned by a datapoint query.
</ParamField>

## Returns

A map of datapoint fields, or an error if the raw datapoint cannot be decoded.

<RequestExample>
  ```go Go theme={"system"}
  descriptor, err := datasets.NewDatapointDescriptor(dataset)
  if err != nil {
      return err
  }

  for data, err := range client.Datapoints.Query(ctx,
      dataset.ID,
      datasets.WithTemporalExtent(queryInterval),
  ) {
      if err != nil {
          return err
      }

      datapoint, err := datasets.UnmarshalDatapoint(descriptor, data)
      if err != nil {
          return err
      }
      fmt.Println(datapoint["id"])
  }
  ```
</RequestExample>


# Datasets.Update
Source: https://docs.tilebox.com/api-reference/go/datasets/Update



```go theme={"system"}
func (datasetClient) Update(
    ctx context.Context,
    id uuid.UUID,
    kind datasets.DatasetKind,
    codeName string,
    name string,
    fields []datasets.Field,
    options ...datasets.DatasetOption,
) (*datasets.Dataset, error)
```

Update an existing dataset by ID with the given code name, display name, schema kind, custom fields, and metadata.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the dataset to update.
</ParamField>

<ParamField type="datasets.DatasetKind">
  The dataset kind.
</ParamField>

<ParamField type="string">
  The stable code identifier for the dataset.
</ParamField>

<ParamField type="string">
  The display name of the dataset.
</ParamField>

<ParamField type="[]datasets.Field">
  The full custom field list for the dataset schema.
</ParamField>

<ParamField type="[]datasets.DatasetOption">
  Options for dataset metadata.
</ParamField>

## Options

<ParamField>
  Set a short dataset summary.
</ParamField>

<ParamField>
  Set the dataset's markdown description.
</ParamField>

## Returns

The updated dataset object.

<RequestExample>
  ```go Go theme={"system"}
  dataset, err := client.Datasets.Update(ctx,
      datasetID,
      datasets.KindSpatiotemporal,
      "my_catalog",
      "My catalog",
      []datasets.Field{
          field.String("source").Description("Source system"),
          field.Float64("cloud_cover"),
          field.Timestamp("processed_at"),
      },
      datasets.WithDescription("Catalog of scenes prepared for analysis."),
  )
  ```
</RequestExample>


# Automations.Get
Source: https://docs.tilebox.com/api-reference/go/workflows/Automations.Get



```go theme={"system"}
func (automationClient) Get(
    ctx context.Context,
    automationID uuid.UUID,
) (*workflows.Automation, error)
```

Get an automation prototype by ID.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the automation.
</ParamField>

## Returns

An automation object.

<RequestExample>
  ```go Go theme={"system"}
  automation, err := client.Automations.Get(ctx, automationID)
  ```
</RequestExample>


# Automations.GetStorageLocation
Source: https://docs.tilebox.com/api-reference/go/workflows/Automations.GetStorageLocation



```go theme={"system"}
func (automationClient) GetStorageLocation(
    ctx context.Context,
    storageLocationID uuid.UUID,
) (*workflows.StorageLocation, error)
```

Get a storage location used by automation storage event triggers.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the storage location.
</ParamField>

## Returns

A storage location object.

<RequestExample>
  ```go Go theme={"system"}
  location, err := client.Automations.GetStorageLocation(ctx, storageLocationID)
  ```
</RequestExample>


# Automations.List
Source: https://docs.tilebox.com/api-reference/go/workflows/Automations.List



```go theme={"system"}
func (automationClient) List(ctx context.Context) ([]*workflows.Automation, error)
```

List all automation prototypes.

## Returns

A list of automation objects.

<RequestExample>
  ```go Go theme={"system"}
  automations, err := client.Automations.List(ctx)
  ```
</RequestExample>


# Automations.ListStorageLocations
Source: https://docs.tilebox.com/api-reference/go/workflows/Automations.ListStorageLocations



```go theme={"system"}
func (automationClient) ListStorageLocations(ctx context.Context) ([]*workflows.StorageLocation, error)
```

List storage locations available for automation storage event triggers.

## Returns

A list of storage location objects.

<RequestExample>
  ```go Go theme={"system"}
  locations, err := client.Automations.ListStorageLocations(ctx)
  ```
</RequestExample>


# Clusters.Create
Source: https://docs.tilebox.com/api-reference/go/workflows/Clusters.Create



```go theme={"system"}
func (*ClusterClient) Create(
    ctx context.Context,
    name string,
    options ...cluster.CreateOption,
) (*workflows.Cluster, error)
```

Create a cluster.

## Parameters

<ParamField type="string">
  A display name for the cluster.
</ParamField>

<ParamField type="cluster.CreateOption">
  Optional cluster description.
</ParamField>

<ParamField type="cluster.CreateOption">
  Optional cluster slug. If omitted, Tilebox generates a slug from the cluster name.
</ParamField>

## Returns

The created cluster object, including its slug, name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```go Go theme={"system"}
  createdCluster, err := client.Clusters.Create(ctx,
      "My cluster",
      cluster.WithDescription("Development workloads"),
      cluster.WithSlug("dev"),
  )
  ```
</RequestExample>


# Clusters.Delete
Source: https://docs.tilebox.com/api-reference/go/workflows/Clusters.Delete



```go theme={"system"}
func (*ClusterClient) Delete(ctx context.Context, slug string) error
```

Delete a cluster by its slug.

## Parameters

<ParamField type="string">
  The slug of the cluster to delete
</ParamField>

## Returns

An error if the cluster could not be deleted.

<RequestExample>
  ```go Go theme={"system"}
  err := client.Clusters.Delete(ctx,
      "my-cluster-tZD9Ca1qsqt3V"
  )
  ```
</RequestExample>

## Errors

<ParamField type="cluster not found">
  The specified cluster does not exist.
</ParamField>


# Clusters.Get
Source: https://docs.tilebox.com/api-reference/go/workflows/Clusters.Get



```go theme={"system"}
func (*ClusterClient) Get(
    ctx context.Context,
    slug string,
) (*workflows.Cluster, error)
```

Get a cluster by its slug.

## Parameters

<ParamField type="string">
  The slug of the cluster
</ParamField>

## Returns

A cluster object, including its slug, name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```go Go theme={"system"}
  cluster, err := client.Clusters.Get(ctx,
      "my-cluster-tZD9Ca1qsqt3V"
  )
  ```
</RequestExample>

## Errors

<ParamField type="cluster not found">
  The specified cluster does not exist.
</ParamField>


# Clusters.List
Source: https://docs.tilebox.com/api-reference/go/workflows/Clusters.List



```go theme={"system"}
func (*ClusterClient) List(ctx context.Context) ([]*workflows.Cluster, error)
```

Fetch all available clusters.

## Returns

A list of all available clusters. Each cluster includes its slug, name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```go Go theme={"system"}
  clusters, err := client.Clusters.List(ctx)
  ```
</RequestExample>


# Clusters.Update
Source: https://docs.tilebox.com/api-reference/go/workflows/Clusters.Update



```go theme={"system"}
func (*ClusterClient) Update(
    ctx context.Context,
    slug string,
    options ...cluster.UpdateOption,
) (*workflows.Cluster, error)
```

Update a cluster.

## Parameters

<ParamField type="string">
  The slug of the cluster to update.
</ParamField>

<ParamField type="cluster.UpdateOption">
  Optional new display name for the cluster.
</ParamField>

<ParamField type="cluster.UpdateOption">
  Optional new cluster description. Pass an empty string to clear the description.
</ParamField>

## Returns

The updated cluster object, including its slug, name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```go Go theme={"system"}
  updatedCluster, err := client.Clusters.Update(ctx,
      "dev",
      cluster.WithName("Development"),
      cluster.WithDescription("Development workloads"),
  )
  ```
</RequestExample>

## Errors

<ParamField type="cluster not found">
  The specified cluster does not exist.
</ParamField>


# Collect
Source: https://docs.tilebox.com/api-reference/go/workflows/Collect



```go theme={"system"}
func Collect[K any](seq iter.Seq2[K, error]) ([]K, error)
```

Convert any sequence into a slice.

It returns an error if any element in the sequence has a non-nil error.

## Parameters

<ParamField type="iter.Seq2[K, error]">
  The sequence to convert.
</ParamField>

## Returns

A slice of `K` or an error if any.

<RequestExample>
  ```go Go theme={"system"}
  jobs, err := workflows.Collect(
    client.Jobs.Query(ctx, job.WithTemporalExtent(interval)),
  )
  ```
</RequestExample>


# workflows.ConfigureConsoleLogging
Source: https://docs.tilebox.com/api-reference/go/workflows/ConfigureConsoleLogging



```go theme={"system"}
func ConfigureConsoleLogging(level slog.Level)
```

Configure the default `slog` logger to write workflow logs to standard output at the given level.

The console handler composes with the Tilebox log exporter installed by `workflows.NewClient()`, so logs can be sent to Tilebox and printed locally. Calling `ConfigureConsoleLogging` more than once does not add duplicate console handlers.

## Parameters

<ParamField type="slog.Level">
  The minimum log level to print to the console.
</ParamField>

## Returns

Nothing.

<RequestExample>
  ```go Go theme={"system"}
  import (
      "context"
      "log/slog"

      "github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
      ctx := context.Background()
      workflows.ConfigureConsoleLogging(slog.LevelDebug)

      client := workflows.NewClient()
      runner, err := client.NewTaskRunner(ctx)
      if err != nil {
          slog.ErrorContext(ctx, "failed to create runner", slog.Any("error", err))
          return
      }

      if err := runner.RunForever(ctx); err != nil {
          slog.ErrorContext(ctx, "runner failed", slog.Any("error", err))
      }
  }
  ```
</RequestExample>


# workflows.DefaultProgress
Source: https://docs.tilebox.com/api-reference/go/workflows/DefaultProgress



```go theme={"system"}
func DefaultProgress() workflows.ProgressTracker
```

Return the default progress tracker for the currently executing task.

Use the returned tracker to update total and completed work units inside a task `Execute` method.

## Returns

A progress tracker for the default progress indicator.

<RequestExample>
  ```go Go theme={"system"}
  progress := workflows.DefaultProgress()
  if err := progress.Add(ctx, 10); err != nil {
      return err
  }
  if err := progress.Done(ctx, 1); err != nil {
      return err
  }
  ```
</RequestExample>


# workflows.GetCurrentCluster
Source: https://docs.tilebox.com/api-reference/go/workflows/GetCurrentCluster



```go theme={"system"}
workflows.GetCurrentCluster(ctx context.Context) (string, error)
```

Get the current cluster slug.

This function is intended to be used in tasks.

## Returns

The current cluster slug.

<RequestExample>
  ```go Go theme={"system"}
  type Task struct{}

  func (t *Task) Execute(ctx context.Context) error {
  	clusterSlug, err := workflows.GetCurrentCluster(ctx)
  	if err != nil {
  		return fmt.Errorf("failed to get current cluster: %w", err)
  	}

  	return nil
  }
  ```
</RequestExample>


# Jobs.Cancel
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.Cancel



```go theme={"system"}
func (*JobClient) Cancel(ctx context.Context, jobID uuid.UUID) error
```

Cancel a job. When a job is canceled, no queued tasks will be picked up by runners and executed even if runners are idle. Tasks that are already being executed will finish their execution and not be interrupted. All sub-tasks spawned from such tasks after the cancellation will not be picked up by runners.

## Parameters

<ParamField type="uuid.UUID">
  The id of the job
</ParamField>

## Returns

An error if the job could not be cancelled.

<RequestExample>
  ```go Go theme={"system"}
  err := client.Jobs.Cancel(ctx,
      uuid.MustParse("0195c87a-49f6-5ffa-e3cb-92215d057ea6"),
  )
  ```
</RequestExample>


# Jobs.Get
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.Get



```go theme={"system"}
func (*JobClient) Get(
    ctx context.Context,
    jobID uuid.UUID,
) (*workflows.Job, error)
```

Get a job by its id.

## Parameters

<ParamField type="uuid.UUID">
  The id of the job
</ParamField>

## Returns

A job object.

<RequestExample>
  ```go Go theme={"system"}
  job, err := client.Jobs.Get(ctx,
      uuid.MustParse("0195c87a-49f6-5ffa-e3cb-92215d057ea6"),
  )
  ```
</RequestExample>

## Errors

<ParamField type="job not found">
  The specified job does not exist.
</ParamField>


# Jobs.Query
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.Query



```go theme={"system"}
func (*JobClient) Query(
    ctx context.Context,
    options ...job.QueryOption,
) iter.Seq2[*workflows.Job, error]
```

Query jobs matching the provided options.

The jobs are lazily loaded across pages and returned as a sequence of jobs.
The output sequence can be transformed into a slice of Job using [Collect](/api-reference/go/workflows/Collect) function.

## Parameters

<ParamField type="[]job.QueryOption">
  Options for querying jobs
</ParamField>

## Options

<ParamField>
  Filter jobs by time or job ID interval.
</ParamField>

<ParamField>
  Filter jobs by the automations that submitted them.
</ParamField>

<ParamField>
  Filter jobs by job state.
</ParamField>

<ParamField>
  Filter jobs by name.
</ParamField>

<ParamField>
  Filter jobs by the states of their tasks. Only jobs that have at least one task in any of the given states will be returned. Useful for finding jobs with [optional](/workflows/concepts/tasks#optional-tasks) task failures.
</ParamField>

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the total number of jobs yielded by the sequence.
</ParamField>

<ParamField>
  Sort jobs by submission date. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A sequence of jobs.

<RequestExample>
  ```go Go theme={"system"}
  import (
    "time"
    workflows "github.com/tilebox/tilebox-go/workflows/v1"
    "github.com/tilebox/tilebox-go/workflows/v1/job"
  	"github.com/tilebox/tilebox-go/query"
  )

  interval := query.NewTimeInterval(
    time.Now().Add(-24 * time.Hour),
    time.Now(),
  )

  jobs, err := workflows.Collect(
    client.Jobs.Query(ctx,
      job.WithTemporalExtent(interval),
      job.WithJobStates(job.Running, job.Started),
    ),
  )
  ```
</RequestExample>


# Jobs.QueryLogs
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.QueryLogs



```go theme={"system"}
func (*JobClient) QueryLogs(
    ctx context.Context,
    jobID uuid.UUID,
    options ...job.TelemetryQueryOption,
) iter.Seq2[*workflows.LogRecord, error]
```

Query log records emitted while running a job.

The logs are lazily loaded and returned as a sequence of log records. Use [Collect](/api-reference/go/workflows/Collect) to transform the sequence into a slice.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the job to query logs for.
</ParamField>

<ParamField type="[]job.TelemetryQueryOption">
  Options for querying logs.
</ParamField>

## Options

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the total number of log records yielded by the sequence.
</ParamField>

<ParamField>
  Sort logs by time. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A sequence of log records. Each record includes `Time`, `SeverityText`, `Body`, trace IDs, span IDs, and structured attributes.

<RequestExample>
  ```go Go theme={"system"}
  import (
      "fmt"
      "log/slog"
      "time"

      "github.com/google/uuid"
      "github.com/tilebox/tilebox-go/workflows/v1"
      "github.com/tilebox/tilebox-go/workflows/v1/job"
  )

  jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  for record, err := range client.Jobs.QueryLogs(
      ctx,
      jobID,
      job.WithSortDirection(job.Ascending),
  ) {
      if err != nil {
          slog.ErrorContext(ctx, "failed to query job logs", slog.Any("error", err))
          return
      }

      fmt.Printf("%s %-5s %s\n",
          record.Time.Format(time.RFC3339),
          record.SeverityText,
          record.Body,
      )
  }
  ```
</RequestExample>


# Jobs.QueryLogsPage
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.QueryLogsPage



```go theme={"system"}
func (jobClient) QueryLogsPage(
    ctx context.Context,
    jobID uuid.UUID,
    options ...job.TelemetryQueryOption,
) (*workflows.LogPage, error)
```

Query a single page of log records emitted while running a job.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the job to query logs for.
</ParamField>

<ParamField type="[]job.TelemetryQueryOption">
  Options for querying logs.
</ParamField>

## Options

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the number of log records returned in this page.
</ParamField>

<ParamField>
  Sort logs by time. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A `LogPage` with log records and an optional `NextCursor` for the next page.

<RequestExample>
  ```go Go theme={"system"}
  page, err := client.Jobs.QueryLogsPage(ctx,
      jobID,
      job.WithLimit(100),
      job.WithSortDirection(job.Ascending),
  )
  ```
</RequestExample>


# Jobs.QueryPage
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.QueryPage



```go theme={"system"}
func (jobClient) QueryPage(
    ctx context.Context,
    options ...job.QueryOption,
) (*workflows.JobPage, error)
```

Query a single page of jobs matching the provided options.

Use `QueryPage` when you need manual pagination. Use [`Jobs.Query`](/api-reference/go/workflows/Jobs.Query) for automatic lazy pagination.

## Parameters

<ParamField type="[]job.QueryOption">
  Options for querying jobs.
</ParamField>

## Options

<ParamField>
  Filter jobs by time or job ID interval.
</ParamField>

<ParamField>
  Filter jobs by the automations that submitted them.
</ParamField>

<ParamField>
  Filter jobs by job state.
</ParamField>

<ParamField>
  Filter jobs by task state. Only jobs with at least one task in any of the given states are returned.
</ParamField>

<ParamField>
  Filter jobs by name.
</ParamField>

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the number of jobs returned in this page.
</ParamField>

<ParamField>
  Sort jobs by submission date. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A `JobPage` with jobs and an optional `NextCursor` for the next page.

<RequestExample>
  ```go Go theme={"system"}
  page, err := client.Jobs.QueryPage(ctx,
      job.WithJobStates(job.Running, job.Started),
      job.WithLimit(50),
      job.WithSortDirection(job.Descending),
  )
  if err != nil {
      return err
  }

  if page.NextCursor != nil {
      nextPage, err := client.Jobs.QueryPage(ctx,
          job.WithCursor(page.NextCursor),
          job.WithLimit(50),
      )
      _ = nextPage
      _ = err
  }
  ```
</RequestExample>


# Jobs.QuerySpans
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.QuerySpans



```go theme={"system"}
func (*JobClient) QuerySpans(
    ctx context.Context,
    jobID uuid.UUID,
    options ...job.TelemetryQueryOption,
) iter.Seq2[*workflows.Span, error]
```

Query spans emitted while running a job.

The spans are lazily loaded and returned as a sequence of spans. Use [Collect](/api-reference/go/workflows/Collect) to transform the sequence into a slice.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the job to query spans for.
</ParamField>

<ParamField type="[]job.TelemetryQueryOption">
  Options for querying spans.
</ParamField>

## Options

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the total number of spans yielded by the sequence.
</ParamField>

<ParamField>
  Sort spans by start time. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A sequence of spans. Each span includes `StartTime`, `Name`, `StatusCode`, `Attributes`, and `Duration()`.

<RequestExample>
  ```go Go theme={"system"}
  import (
      "fmt"
      "log/slog"
      "time"

      "github.com/google/uuid"
      "github.com/tilebox/tilebox-go/workflows/v1"
      "github.com/tilebox/tilebox-go/workflows/v1/job"
  )

  jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  for span, err := range client.Jobs.QuerySpans(
      ctx,
      jobID,
      job.WithSortDirection(job.Ascending),
  ) {
      if err != nil {
          slog.ErrorContext(ctx, "failed to query job spans", slog.Any("error", err))
          return
      }

      fmt.Printf("%s %-40s %s\n",
          span.StartTime.Format(time.RFC3339),
          span.Name,
          span.Duration(),
      )
  }
  ```
</RequestExample>


# Jobs.QuerySpansPage
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.QuerySpansPage



```go theme={"system"}
func (jobClient) QuerySpansPage(
    ctx context.Context,
    jobID uuid.UUID,
    options ...job.TelemetryQueryOption,
) (*workflows.SpanPage, error)
```

Query a single page of spans emitted while running a job.

## Parameters

<ParamField type="uuid.UUID">
  The ID of the job to query spans for.
</ParamField>

<ParamField type="[]job.TelemetryQueryOption">
  Options for querying spans.
</ParamField>

## Options

<ParamField>
  Start the query after the cursor returned by a previous page.
</ParamField>

<ParamField>
  Limit the number of spans returned in this page.
</ParamField>

<ParamField>
  Sort spans by start time. Use `job.Ascending` for oldest first or `job.Descending` for newest first.
</ParamField>

## Returns

A `SpanPage` with spans and an optional `NextCursor` for the next page.

<RequestExample>
  ```go Go theme={"system"}
  page, err := client.Jobs.QuerySpansPage(ctx,
      jobID,
      job.WithLimit(100),
      job.WithSortDirection(job.Ascending),
  )
  ```
</RequestExample>


# Jobs.Retry
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.Retry



```go theme={"system"}
func (*JobClient) Retry(
    ctx context.Context,
    jobID uuid.UUID,
) (int64, error)
```

Retry a job. All failed tasks will become queued again, and queued tasks will be picked up by runners again.

## Parameters

<ParamField type="uuid.UUID">
  The id of the job to retry
</ParamField>

## Returns

The number of tasks that were rescheduled.

<RequestExample>
  ```go Go theme={"system"}
  nbRescheduled, err := client.Jobs.Retry(ctx,
      uuid.MustParse("0195c87a-49f6-5ffa-e3cb-92215d057ea6"),
  )
  ```
</RequestExample>

## Errors

<ParamField type="job not found">
  The specified job does not exist.
</ParamField>


# Jobs.Submit
Source: https://docs.tilebox.com/api-reference/go/workflows/Jobs.Submit



```go theme={"system"}
func (*JobClient) Submit(
    ctx context.Context,
    jobName string,
    tasks []workflows.Task,
    options ...job.SubmitOption
) (*workflows.Job, error)
```

Submit a job.

## Parameters

<ParamField type="string">
  The name of the job
</ParamField>

<ParamField type="[]Task">
  The root task for the job. This task is executed first and can submit subtasks to manage the entire workflow. A job can have optionally consist of multiple root tasks.
</ParamField>

<ParamField type="[]job.SubmitOption">
  Options for the job
</ParamField>

## Options

<ParamField>
  Set the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtask in case it fails
</ParamField>

<ParamField>
  The [cluster](/workflows/concepts/clusters#managing-clusters) to run the root task on. If not provided, the default cluster is used.
</ParamField>

## Returns

A job object.

<RequestExample>
  ```go Go theme={"system"}
  job, err := client.Jobs.Submit(ctx,
      "My job",
      []workflows.Task{rootTask},
  )
  ```
</RequestExample>


# Client.NewPollingTaskRunner
Source: https://docs.tilebox.com/api-reference/go/workflows/NewPollingTaskRunner



```go theme={"system"}
func (*Client) NewPollingTaskRunner(
    ctx context.Context,
    executor workflows.TaskExecutor,
    options ...runner.Option,
) (*workflows.PollingTaskRunner, error)
```

Create a polling runner for a custom task executor.

Use this lower-level runner when you need a custom execution backend, such as a dynamic runtime, but still want the Tilebox polling protocol to request, lease, and report tasks.

## Parameters

<ParamField type="workflows.TaskExecutor">
  The executor that reports task capabilities and executes leased tasks.
</ParamField>

<ParamField type="[]runner.Option">
  Options for initializing the polling runner.
</ParamField>

## Options

<ParamField>
  The cluster to poll for tasks. If not provided, the default cluster is used.
</ParamField>

<ParamField>
  Set the logger to use for the polling runner.
</ParamField>

## Returns

The created polling runner.

<RequestExample>
  ```go Go theme={"system"}
  runner, err := client.NewPollingTaskRunner(ctx, executor)
  ```
</RequestExample>


# Client.NewTaskRunner
Source: https://docs.tilebox.com/api-reference/go/workflows/NewTaskRunner



```go theme={"system"}
func (*Client) NewTaskRunner(
  ctx context.Context,
  options ...runner.Option,
) (*workflows.TaskRunner, error)
```

Initialize a runner.

## Parameters

<ParamField type="[]runner.Option">
  Options for initializing the runner
</ParamField>

## Options

<ParamField>
  The [cluster](/workflows/concepts/clusters#managing-clusters) to connect to. If not provided, the default cluster is used.
</ParamField>

<ParamField>
  Set the logger to use for the runner
</ParamField>

<ParamField>
  Disable OpenTelemetry metrics for the runner
</ParamField>

## Returns

The created runner object.

<RequestExample>
  ```go Go theme={"system"}
  runner, err := client.NewTaskRunner(ctx)
  ```
</RequestExample>


# PollingTaskRunner.HasActiveTask
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.HasActiveTask



```go theme={"system"}
func (*PollingTaskRunner) HasActiveTask() bool
```

Report whether the polling runner currently has an active task.

## Returns

`true` if a task is currently active.

<RequestExample>
  ```go Go theme={"system"}
  if runner.HasActiveTask() {
      slog.Info("runner has an active task")
  }
  ```
</RequestExample>


# PollingTaskRunner.InterruptActiveTask
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.InterruptActiveTask



```go theme={"system"}
func (*PollingTaskRunner) InterruptActiveTask(ctx context.Context) error
```

Interrupt the active task by canceling its lease extension and reporting it as failed.

If there is no active task, the method returns nil.

## Parameters

<ParamField type="context.Context">
  The context used to report the interrupted task.
</ParamField>

## Returns

An error if the active task could not be reported as failed.

<RequestExample>
  ```go Go theme={"system"}
  if err := runner.InterruptActiveTask(ctx); err != nil {
      return err
  }
  ```
</RequestExample>


# PollingTaskRunner.IsRequestingTasks
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.IsRequestingTasks



```go theme={"system"}
func (*PollingTaskRunner) IsRequestingTasks() bool
```

Report whether the polling runner is currently requesting new tasks.

## Returns

`true` if the runner is requesting new tasks.

<RequestExample>
  ```go Go theme={"system"}
  if runner.IsRequestingTasks() {
      slog.Info("runner is requesting tasks")
  }
  ```
</RequestExample>


# PollingTaskRunner.RunAll
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.RunAll



```go theme={"system"}
func (*PollingTaskRunner) RunAll(ctx context.Context) error
```

Run the polling runner until there are no more tasks available.

## Parameters

<ParamField type="context.Context">
  The context controlling the polling loop lifetime.
</ParamField>

## Returns

An error if the polling loop fails.

<RequestExample>
  ```go Go theme={"system"}
  if err := runner.RunAll(ctx); err != nil {
      return err
  }
  ```
</RequestExample>


# PollingTaskRunner.RunForever
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.RunForever



```go theme={"system"}
func (*PollingTaskRunner) RunForever(ctx context.Context) error
```

Run the polling runner continuously, polling for tasks when idle.

## Parameters

<ParamField type="context.Context">
  The context controlling the polling loop lifetime.
</ParamField>

## Returns

An error if the polling loop fails.

<RequestExample>
  ```go Go theme={"system"}
  if err := runner.RunForever(ctx); err != nil {
      return err
  }
  ```
</RequestExample>


# PollingTaskRunner.StopRequestingNewTasks
Source: https://docs.tilebox.com/api-reference/go/workflows/PollingTaskRunner.StopRequestingNewTasks



```go theme={"system"}
func (*PollingTaskRunner) StopRequestingNewTasks()
```

Stop requesting new tasks from the Tilebox API.

The runner can still finish and report an active task after this method is called.

## Returns

Nothing.

<RequestExample>
  ```go Go theme={"system"}
  runner.StopRequestingNewTasks()
  ```
</RequestExample>


# workflows.Progress
Source: https://docs.tilebox.com/api-reference/go/workflows/Progress



```go theme={"system"}
func Progress(label string) workflows.ProgressTracker
```

Return a named progress tracker for the currently executing task.

Use named trackers when a task reports multiple progress indicators.

## Parameters

<ParamField type="string">
  The progress indicator label.
</ParamField>

## Returns

A progress tracker for the named progress indicator.

<RequestExample>
  ```go Go theme={"system"}
  download := workflows.Progress("Download")
  if err := download.Add(ctx, 100); err != nil {
      return err
  }
  ```
</RequestExample>


# ProgressTracker.Add
Source: https://docs.tilebox.com/api-reference/go/workflows/ProgressTracker.Add



```go theme={"system"}
func (ProgressTracker) Add(ctx context.Context, n uint64) error
```

Add total work units to a progress indicator.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="uint64">
  The number of total work units to add.
</ParamField>

## Returns

An error if the context is not a task execution context.

<RequestExample>
  ```go Go theme={"system"}
  progress := workflows.DefaultProgress()
  err := progress.Add(ctx, 10)
  ```
</RequestExample>


# ProgressTracker.Done
Source: https://docs.tilebox.com/api-reference/go/workflows/ProgressTracker.Done



```go theme={"system"}
func (ProgressTracker) Done(ctx context.Context, n uint64) error
```

Mark work units as completed for a progress indicator.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="uint64">
  The number of completed work units to add.
</ParamField>

## Returns

An error if the context is not a task execution context.

<RequestExample>
  ```go Go theme={"system"}
  progress := workflows.DefaultProgress()
  err := progress.Done(ctx, 1)
  ```
</RequestExample>


# workflows.SetTaskDisplay
Source: https://docs.tilebox.com/api-reference/go/workflows/SetTaskDisplay



```go theme={"system"}
func SetTaskDisplay(ctx context.Context, display string) error
```

Set the display label of the currently executing task.

This function is intended to be used inside a task `Execute` method.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="string">
  The display label to set for the current task.
</ParamField>

## Returns

An error if the context is not a task execution context.

<RequestExample>
  ```go Go theme={"system"}
  func (t *ProcessScene) Execute(ctx context.Context) error {
      if err := workflows.SetTaskDisplay(ctx, "Process scene " + t.SceneID); err != nil {
          return err
      }
      return nil
  }
  ```
</RequestExample>


# workflows.SubmitSubtask
Source: https://docs.tilebox.com/api-reference/go/workflows/SubmitSubtask



```go theme={"system"}
workflows.SubmitSubtask(
	ctx context.Context,
	task workflows.Task,
	options ...subtask.SubmitOption,
) (subtask.FutureTask, error)
```

Submit a subtask to the runner.

This function is intended to be used in tasks.

## Parameters

<ParamField type="Task">
  A subtask to submit
</ParamField>

<ParamField type="[]subtask.SubmitOption">
  Options for the subtask
</ParamField>

## Options

<ParamField>
  Set dependencies for the task
</ParamField>

<ParamField>
  Set the cluster slug of the cluster where the task will be executed.
</ParamField>

<ParamField>
  Set the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtask in case it fails
</ParamField>

<ParamField>
  Mark the subtask as [optional](/workflows/concepts/tasks#optional-tasks). An optional subtask will not fail the job if it fails. Tasks that depend on it will still execute even if it failed.
</ParamField>

## Returns

A future task that can be used to set dependencies between tasks.

<RequestExample>
  ```go Go theme={"system"}

  type MySubTask struct {
  	Sensor string
  	Value  float64
  }

  type Task struct{}

  func (t *Task) Execute(ctx context.Context) error {
  	err := workflows.SubmitSubtask(ctx, 
  		&MySubTask{
  			Sensor: "A",
  			Value:  42,
  		},
  	)
  	if err != nil {
  		return fmt.Errorf("failed to submit subtasks: %w", err)
  	}

  	return nil
  }
  ```
</RequestExample>


# workflows.SubmitSubtasks
Source: https://docs.tilebox.com/api-reference/go/workflows/SubmitSubtasks



```go theme={"system"}
workflows.SubmitSubtasks(
	ctx context.Context,
	tasks []workflows.Task,
	options ...subtask.SubmitOption,
) ([]subtask.FutureTask, error)
```

Submit multiple subtasks to the runner. Same as [SubmitSubtask](/api-reference/go/workflows/SubmitSubtask), but accepts a list of tasks.

This function is intended to be used in tasks.

## Parameters

<ParamField type="[]Task">
  A list of tasks to submit
</ParamField>

<ParamField type="[]subtask.SubmitOption">
  Options for the subtasks
</ParamField>

## Options

<ParamField>
  Set dependencies for the tasks
</ParamField>

<ParamField>
  Set the cluster slug of the cluster where the tasks will be executed.
</ParamField>

<ParamField>
  Set the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtasks in case it fails
</ParamField>

<ParamField>
  Mark the subtasks as [optional](/workflows/concepts/tasks#optional-tasks). Optional subtasks will not fail the job if they fail. Tasks that depend on them will still execute even if they failed.
</ParamField>

## Returns

A list of future tasks that can be used to set dependencies between tasks.

<RequestExample>
  ```go Go theme={"system"}

  type MySubTask struct {
  	Sensor string
  	Value  float64
  }

  type Task struct{}

  func (t *Task) Execute(ctx context.Context) error {
  	err := workflows.SubmitSubtasks(ctx, 
  		[]workflows.Task{
  			&MySubTask{
  				Sensor: "A",
  				Value:  42,
  			},
  			&MySubTask{
  				Sensor: "B",
  				Value:  42,
  			}
  		},
  	)
  	if err != nil {
  		return fmt.Errorf("failed to submit subtasks: %w", err)
  	}

  	return nil
  }
  ```
</RequestExample>


# Task
Source: https://docs.tilebox.com/api-reference/go/workflows/Task



```go theme={"system"}
type Task interface{}
```

Base interface for Tilebox workflows [tasks](/workflows/concepts/tasks).
It doesn't need to be identifiable or executable, but it can be both (see below).

## Methods

```go theme={"system"}
Task.Execute(ctx context.Context) error
```

The entry point for the execution of the task.
If not defined, the task can't be registered with a runner but can still be submitted.

```go theme={"system"}
Task.Identifier() TaskIdentifier
```

Provides a user-defined task identifier.
The identifier is used to uniquely identify the task and specify its version.
If not defined, the runner will generate an identifier for it using reflection.

## JSON-serializable task

```go theme={"system"}
type SampleTask struct {
	Message      string
	Depth        int
	BranchFactor int
}
```

Optional task [input parameters](/workflows/concepts/tasks#input-parameters), defined as struct fields.
Supported types are all types supported by [json.Marshal](https://pkg.go.dev/encoding/json#Marshal).

## Protobuf-serializable task

```go theme={"system"}
type SampleTask struct {
	examplesv1.SpawnWorkflowTreeTask
}
```

Task can also be defined as a protobuf message.
An example using task protobuf messages can be found [here](https://github.com/tilebox/tilebox-go/tree/main/examples/sampleworkflow).

<RequestExample>
  ```go Go theme={"system"}
  package helloworld

  import (
  	"context"
  	"fmt"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type MyFirstTask struct{}

  func (t *MyFirstTask) Execute(ctx context.Context) error {
  	fmt.Println("Hello World!")
  	return nil
  }

  func (t *MyFirstTask) Identifier() workflows.TaskIdentifier {
  	return workflows.NewTaskIdentifier("tilebox.workflows.MyTask", "v3.2")
  }

  type MyFirstParameterizedTask struct {
  	Name    string
  	Greet   bool
  	Data    map[string]string
  }

  func (t *MyFirstParameterizedTask) Execute(ctx context.Context) error {
  	if t.Greet {
  		fmt.Printf("Hello %s!\n", t.Name)
  	}
  	return nil
  }
  ```
</RequestExample>


# TaskExecutor
Source: https://docs.tilebox.com/api-reference/go/workflows/TaskExecutor



```go theme={"system"}
type TaskExecutor interface {
    TaskIdentifiers() []*workflowsv1.TaskIdentifier
    ExecuteTask(ctx context.Context, task *workflowsv1.Task) (*workflowsv1.ExecuteTaskResponse, error)
}
```

`TaskExecutor` is the interface used by [`NewPollingTaskRunner`](/api-reference/go/workflows/NewPollingTaskRunner) for custom task execution backends.

Use this interface when you need Tilebox to poll and lease tasks, but you want another runtime to execute them.

## Methods

<ParamField type="func() []*workflowsv1.TaskIdentifier">
  Return the task implementations the executor can currently run. This method should be cheap and non-blocking.
</ParamField>

<ParamField type="func(context.Context, *workflowsv1.Task) (*workflowsv1.ExecuteTaskResponse, error)">
  Execute a leased task and return the computed or failed task response.
</ParamField>


# TaskRunner.GetRegisteredTask
Source: https://docs.tilebox.com/api-reference/go/workflows/TaskRunner.GetRegisteredTask



```go theme={"system"}
func (*TaskRunner) GetRegisteredTask(
  identifier workflows.TaskIdentifier,
) (workflows.ExecutableTask, bool)
```

Get the task with the given identifier.

## Parameters

<ParamField type="TaskIdentifier">
  The identifier of the registered task.
</ParamField>

## Returns

The registered task. Returns `false` if not found.

<RequestExample>
  ```go Go theme={"system"}
  identifier := workflows.NewTaskIdentifier("my-task", "v1.0")

  task, found := runner.GetRegisteredTask(
      identifier,
  )
  ```
</RequestExample>


# TaskRunner.RegisterTasks
Source: https://docs.tilebox.com/api-reference/go/workflows/TaskRunner.RegisterTasks



```go theme={"system"}
func (*TaskRunner) RegisterTasks(tasks ...workflows.ExecutableTask) error
```

Register tasks that can be executed by this runner.

## Parameters

<ParamField type="[]ExecutableTask">
  A list of task classes that this runner can execute
</ParamField>

## Returns

An error if the tasks could not be registered.

<RequestExample>
  ```go Go theme={"system"}
  err := runner.RegisterTasks(
      &MyTask{},
      &MyOtherTask{},
  )
  ```
</RequestExample>


# TaskRunner.RunAll
Source: https://docs.tilebox.com/api-reference/go/workflows/TaskRunner.RunAll



```go theme={"system"}
func (*TaskRunner) RunAll(ctx context.Context) error
```

Run the runner until there are no more tasks available.

Use `RunAll` for finite worker processes and local draining workflows. Use [`RunForever`](/api-reference/go/workflows/TaskRunner.RunForever) for long-running runners.

## Parameters

<ParamField type="context.Context">
  The context controlling the runner lifetime.
</ParamField>

## Returns

An error if the polling loop fails.

<RequestExample>
  ```go Go theme={"system"}
  if err := runner.RunAll(ctx); err != nil {
      return err
  }
  ```
</RequestExample>


# TaskRunner.RunForever
Source: https://docs.tilebox.com/api-reference/go/workflows/TaskRunner.RunForever



```go theme={"system"}
func (*TaskRunner) RunForever(ctx context.Context) error
```

Run the runner continuously, polling for tasks when idle.

`RunForever` stops when the context is canceled or when the process receives a supported shutdown signal.

## Parameters

<ParamField type="context.Context">
  The context controlling the runner lifetime.
</ParamField>

## Returns

An error if the polling loop fails.

<RequestExample>
  ```go Go theme={"system"}
  if err := runner.RunForever(ctx); err != nil {
      return err
  }
  ```
</RequestExample>


# workflows.WithSpan
Source: https://docs.tilebox.com/api-reference/go/workflows/WithSpan



```go theme={"system"}
func WithSpan(
    ctx context.Context,
    name string,
    f func(ctx context.Context) error,
) error
```

Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer.

Use `WithSpan` inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="string">
  The name of the span.
</ParamField>

<ParamField type="func(context.Context) error">
  The function to wrap.
</ParamField>

## Returns

The error returned by `f`, if any.

<RequestExample>
  ```go Go theme={"system"}
  import (
      "context"
      "fmt"

      "github.com/tilebox/tilebox-go/workflows/v1"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
      err := workflows.WithSpan(ctx, "write-output", func(ctx context.Context) error {
          // Write output here.
          return nil
      })
      if err != nil {
          return fmt.Errorf("failed to write output: %w", err)
      }

      return nil
  }
  ```
</RequestExample>


# workflows.WithSpanResult
Source: https://docs.tilebox.com/api-reference/go/workflows/WithSpanResult



```go theme={"system"}
func WithSpanResult[Result any](
    ctx context.Context,
    name string,
    f func(ctx context.Context) (Result, error),
) (Result, error)
```

Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer and return the function result.

Use `WithSpanResult` inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="string">
  The name of the span.
</ParamField>

<ParamField type="func(context.Context) (Result, error)">
  The function to wrap.
</ParamField>

## Returns

The result and error returned by `f`.

<RequestExample>
  ```go Go theme={"system"}
  import (
      "context"
      "fmt"
      "log/slog"

      "github.com/tilebox/tilebox-go/workflows/v1"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
      pixels, err := workflows.WithSpanResult(ctx, "compute-index", func(ctx context.Context) (int, error) {
          // Compute an index here.
          return 42, nil
      })
      if err != nil {
          return fmt.Errorf("failed to compute index: %w", err)
      }

      slog.InfoContext(ctx, "index computed", slog.Int("pixels", pixels))
      return nil
  }
  ```
</RequestExample>


# workflows.WithTaskSpan
Source: https://docs.tilebox.com/api-reference/go/workflows/WithTaskSpan



```go theme={"system"}
workflows.WithTaskSpan(
    ctx context.Context,
    name string,
    f func(ctx context.Context) error,
) error
```

Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer.

`WithTaskSpan` is an alias for [`WithSpan`](/api-reference/go/workflows/WithSpan). Use it inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="string">
  The name of the span.
</ParamField>

<ParamField type="func(context.Context) error">
  The function to wrap.
</ParamField>

## Returns

The error returned by `f`, if any.

<RequestExample>
  ```go Go theme={"system"}
  type Task struct{}

  func (t *Task) Execute(ctx context.Context) error {
  	err := workflows.WithTaskSpan(ctx, "Database insert", func(ctx context.Context) error {
  		// Do something
  		return nil
  	})
    if err != nil {
      return fmt.Errorf("failed to insert into database: %w", err)
    }

  	return nil
  }
  ```
</RequestExample>


# workflows.WithTaskSpanResult
Source: https://docs.tilebox.com/api-reference/go/workflows/WithTaskSpanResult



```go theme={"system"}
workflows.WithTaskSpanResult[Result any](
    ctx context.Context,
    name string,
    f func(ctx context.Context) (Result, error),
) (Result, error)
```

Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer and return the function result.

`WithTaskSpanResult` is an alias for [`WithSpanResult`](/api-reference/go/workflows/WithSpanResult). Use it inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span.

## Parameters

<ParamField type="context.Context">
  The task execution context.
</ParamField>

<ParamField type="string">
  The name of the span.
</ParamField>

<ParamField type="func(context.Context) (Result, error)">
  The function to wrap.
</ParamField>

## Returns

The result and error returned by `f`.

<RequestExample>
  ```go Go theme={"system"}
  type Task struct{}

  func (t *Task) Execute(ctx context.Context) error {
    result, err := workflows.WithTaskSpanResult(ctx, "Expensive Compute", func(ctx context.Context) (int, error) {
  		return 6 * 7, nil
  	})
    if err != nil {
      return fmt.Errorf("failed to compute: %w", err)
    }

  	return nil
  }
  ```
</RequestExample>


# Workflows.Create
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.Create



```go theme={"system"}
func (workflowClient) Create(
    ctx context.Context,
    name string,
    options ...workflows.WorkflowOption,
) (*workflows.Workflow, error)
```

Create a workflow.

## Parameters

<ParamField type="string">
  The workflow display name.
</ParamField>

<ParamField type="[]workflows.WorkflowOption">
  Options for creating the workflow.
</ParamField>

## Options

<ParamField>
  Set the workflow description.
</ParamField>

## Returns

The created workflow object.

<RequestExample>
  ```go Go theme={"system"}
  workflow, err := client.Workflows.Create(ctx,
      "Scene processing",
      workflows.WithDescription("Process new scenes into analysis-ready outputs."),
  )
  ```
</RequestExample>


# Workflows.DeployRelease
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.DeployRelease



```go theme={"system"}
func (workflowClient) DeployRelease(
    ctx context.Context,
    workflowSlug string,
    releaseID uuid.UUID,
    clusterSlugs []string,
) (*workflows.WorkflowReleaseDeployment, error)
```

Deploy a workflow release to one or more clusters.

## Parameters

<ParamField type="string">
  The workflow slug.
</ParamField>

<ParamField type="uuid.UUID">
  The ID of the workflow release to deploy.
</ParamField>

<ParamField type="[]string">
  The clusters to deploy the release to.
</ParamField>

## Returns

The deployment result, including the release and affected clusters.

<RequestExample>
  ```go Go theme={"system"}
  deployment, err := client.Workflows.DeployRelease(ctx,
      workflow.Slug,
      release.ID,
      []string{"production"},
  )
  ```
</RequestExample>


# Workflows.Get
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.Get



```go theme={"system"}
func (workflowClient) Get(
    ctx context.Context,
    slug string,
) (*workflows.Workflow, error)
```

Get a workflow by slug.

## Parameters

<ParamField type="string">
  The workflow slug.
</ParamField>

## Returns

A workflow object.

<RequestExample>
  ```go Go theme={"system"}
  workflow, err := client.Workflows.Get(ctx, "scene-processing")
  ```
</RequestExample>


# Workflows.List
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.List



```go theme={"system"}
func (workflowClient) List(ctx context.Context) ([]*workflows.Workflow, error)
```

List all workflows.

## Returns

A list of workflow objects.

<RequestExample>
  ```go Go theme={"system"}
  workflowsList, err := client.Workflows.List(ctx)
  ```
</RequestExample>


# Workflows.PublishRelease
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.PublishRelease



```go theme={"system"}
func (workflowClient) PublishRelease(
    ctx context.Context,
    workflowSlug string,
    artifactID uuid.UUID,
    content *workflows.ReleaseContent,
) (*workflows.WorkflowRelease, error)
```

Publish an immutable release for a workflow.

## Parameters

<ParamField type="string">
  The workflow slug.
</ParamField>

<ParamField type="uuid.UUID">
  The ID of the release artifact.
</ParamField>

<ParamField type="*workflows.ReleaseContent">
  The files, task identifiers, runner object path, and optional command override included in the release.
</ParamField>

## Returns

The published workflow release, including any clusters where the release is deployed.

<RequestExample>
  ```go Go theme={"system"}
  release, err := client.Workflows.PublishRelease(ctx,
      workflow.Slug,
      artifactID,
      &workflows.ReleaseContent{
          Fingerprint:      fingerprint,
          Tasks:            []workflows.TaskIdentifier{workflows.NewTaskIdentifier("tilebox.com/tasks/ProcessScene", "v1.0")},
          RunnerObjectPath: "worker.runner",
      },
  )
  ```
</RequestExample>


# Workflows.UndeployRelease
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.UndeployRelease



```go theme={"system"}
func (workflowClient) UndeployRelease(
    ctx context.Context,
    workflowSlug string,
    releaseID uuid.UUID,
    clusterSlugs []string,
) (*workflows.WorkflowReleaseDeployment, error)
```

Remove a workflow release from one or more clusters.

## Parameters

<ParamField type="string">
  The workflow slug.
</ParamField>

<ParamField type="uuid.UUID">
  The ID of the workflow release to remove from clusters.
</ParamField>

<ParamField type="[]string">
  The clusters to remove the release from.
</ParamField>

## Returns

The result, including the release and affected clusters.

<RequestExample>
  ```go Go theme={"system"}
  deployment, err := client.Workflows.UndeployRelease(ctx,
      workflow.Slug,
      release.ID,
      []string{"production"},
  )
  ```
</RequestExample>


# Workflows.Update
Source: https://docs.tilebox.com/api-reference/go/workflows/Workflows.Update



```go theme={"system"}
func (workflowClient) Update(
    ctx context.Context,
    slug string,
    options ...workflow.UpdateOption,
) (*workflows.Workflow, error)
```

Update a workflow.

## Parameters

<ParamField type="string">
  The workflow slug.
</ParamField>

<ParamField type="workflow.UpdateOption">
  Optional new display name for the workflow.
</ParamField>

<ParamField type="workflow.UpdateOption">
  Optional new workflow description. Pass an empty string to clear the description.
</ParamField>

## Returns

The updated workflow object.

<RequestExample>
  ```go Go theme={"system"}
  updatedWorkflow, err := client.Workflows.Update(ctx,
      "scene-processing",
      workflow.WithName("Scene processing"),
      workflow.WithDescription("Process new scenes into analysis-ready outputs."),
  )
  ```
</RequestExample>

## Errors

<ParamField type="workflow not found">
  The specified workflow does not exist.
</ParamField>


# Client
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Client



```python theme={"system"}
class Client(
    *,
    url: str = "https://api.tilebox.com",
    token: str | None = None,
    warn_if_unauthenticated: bool = True,
    transport: Literal["grpc", "http1"] = "grpc",
)
```

Create a Tilebox datasets client.

## Parameters

<ParamField type="str">
  Tilebox API Url. Defaults to `https://api.tilebox.com`.
</ParamField>

<ParamField type="str | None">
  The API key to authenticate with. If not set, the `TILEBOX_API_KEY` environment variable is used.
</ParamField>

<ParamField type="bool">
  Whether to log a warning when no API key is provided for the Tilebox API URL. Defaults to `True`.
</ParamField>

<ParamField type="Literal[&#x22;grpc&#x22;, &#x22;http1&#x22;]">
  Network transport to use for API requests. Defaults to `"grpc"`. Use `"http1"` to force the Connect protocol over HTTP/1.1 on networks that do not support gRPC over HTTP/2 correctly.
</ParamField>

If no API key is provided, the client uses anonymous open data access for public datasets.

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  # read token from an environment variable
  client = Client()

  # or provide connection details directly
  client = Client(
      url="https://api.tilebox.com",
      token="YOUR_TILEBOX_API_KEY",
  )
  ```
</RequestExample>


# Client.create_or_update_dataset
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Client.create_or_update_dataset



```python theme={"system"}
def Client.create_or_update_dataset(
    kind: DatasetKind,
    code_name: str,
    fields: list[FieldDict] | None = None,
    *,
    name: str | None = None,
) -> DatasetClient
```

Create a dataset, or update the existing dataset with the same code name.

## Parameters

<ParamField type="DatasetKind">
  The kind of the dataset.
</ParamField>

<ParamField type="str">
  The code name of the dataset.
</ParamField>

<ParamField type="list[FieldDict] | None">
  The custom fields of the dataset. Defaults to an empty field list.
</ParamField>

<ParamField type="str | None">
  The display name of the dataset. Defaults to the code name when creating a dataset, and to the existing name when updating a dataset.
</ParamField>

## Dataset kinds

<ParamField type="DatasetKind">
  A dataset that contains a timestamp field
</ParamField>

<ParamField type="DatasetKind">
  A dataset that contains a timestamp field and a geometry field
</ParamField>

## Field types

<ParamField type="type">
  A string field
</ParamField>

<ParamField type="type">
  A bytes field
</ParamField>

<ParamField type="type">
  A boolean field
</ParamField>

<ParamField type="type">
  A 64-bit signed integer field
</ParamField>

<ParamField type="type">
  A 64-bit unsigned integer field
</ParamField>

<ParamField type="type">
  A 64-bit floating-point number field
</ParamField>

<ParamField type="type">
  A duration field
</ParamField>

<ParamField type="type">
  A timestamp field
</ParamField>

<ParamField type="type">
  A UUID field
</ParamField>

<ParamField type="type">
  A geometry field
</ParamField>

Note that the type can also be a list of one of the types, indicating that the field is an array, e.g. `list[str]`.

## Field options

<ParamField type="str">
  Set the name of the field
</ParamField>

<ParamField type="type">
  Set the type of the field
</ParamField>

<ParamField type="str">
  Set the description of the field to provide more context and details about the data
</ParamField>

<ParamField type="str">
  Set the example value of the field for documentation purposes
</ParamField>

## Returns

A `DatasetClient` for the created or updated dataset.

<RequestExample>
  ```python Python theme={"system"}
  from shapely import Geometry
  from tilebox.datasets import Client
  from tilebox.datasets.data.datasets import DatasetKind

  client = Client()

  dataset = client.create_or_update_dataset(
      kind=DatasetKind.SPATIOTEMPORAL,
      code_name="my_catalog",
      fields=[
          {
              "name": "field1",
              "type": str,
          },
          {
              "name": "field2",
              "type": list[int],
          },
          {
              "name": "field3",
              "type": Geometry,
              "description": "Field 3",
              "example_value": "Value 3",
          },
      ],
      name="My personal catalog",
  )
  ```
</RequestExample>


# Client.dataset
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Client.dataset



```python theme={"system"}
def Client.dataset(slug: str) -> Dataset
```

Get a dataset by its slug.

## Parameters

<ParamField type="string">
  The slug of the dataset
</ParamField>

## Returns

A dataset object.

<RequestExample>
  ```python Python theme={"system"}
  s1_sar = client.dataset(
      "open_data.copernicus.sentinel1_sar"
  )
  ```
</RequestExample>

## Errors

<ParamField type="No such dataset: non.existent.dataset">
  The specified dataset does not exist.
</ParamField>


# Client.datasets
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Client.datasets



```python theme={"system"}
def Client.datasets() -> Datasets
```

Fetch all available datasets.

## Returns

An object containing all available datasets, accessible via dot notation.

<RequestExample>
  ```python Python theme={"system"}
  datasets = client.datasets()

  s1_sar = datasets.open_data.copernicus.sentinel1_sar
  ```
</RequestExample>


# Collection.delete
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Collection.delete



```python theme={"system"}
def Collection.delete(
    datapoints: DatapointIDs,
    *,
    show_progress: bool | Callable[[float], None] = False,
) -> int
```

Delete data points from the collection.

Data points are identified and deleted by their ids.

<Note>
  You need to have write permission on the collection to be able to delete data points.
</Note>

## Parameters

<ParamField type="DatapointIDs">
  Datapoint IDs to delete from the collection.

  Supported `DatapointIDs` types are:

  * A `pandas.DataFrame` containing an `id` column.
  * A `pandas.Series` containing datapoint IDs.
  * An `xarray.Dataset` containing an "id" variable.
  * An `xarray.DataArray` containing datapoint IDs.
  * A `numpy.ndarray` containing datapoint IDs.
  * A `Collection[UUID]` containing datapoint IDs as python built-in `UUID` objects, e.g. `list[UUID]`.
  * A `Collection[str]` containing datapoint IDs as strings, e.g. `list[str]`.
</ParamField>

<ParamField type="bool | Callable[[float], None]">
  If `True`, display a progress bar while deleting many datapoints. You can also pass a callback to receive progress values between `0` and `1`. Defaults to `False`.
</ParamField>

## Returns

The number of data points that were deleted.

<RequestExample>
  ```python Python theme={"system"}
  collection.delete([
    "0195c87a-49f6-5ffa-e3cb-92215d057ea6",
    "0195c87b-bd0e-3998-05cf-af6538f34957",
  ])
  ```
</RequestExample>

## Errors

<ParamField type="No such collection: Non-existent-Collection">
  One of the data points is not found in the collection. If any of the data points are not found,
  nothing will be deleted.
</ParamField>

<ParamField type="Invalid UUID">
  One of the specified ids is not a valid UUID
</ParamField>


# Collection.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Collection.find



```python theme={"system"}
def Collection.find(
    datapoint_id: str,
    skip_data: bool = False
) -> xarray.Dataset
```

Find a specific datapoint in a collection by its id.

## Parameters

<ParamField type="str">
  The id of the datapoint to find.
</ParamField>

<ParamField type="bool">
  If `True`, the response contains only the ID and the timestamp for the datapoint. Defaults to `False`.
</ParamField>

## Returns

An [`xarray.Dataset`](/sdks/python/xarray) containing the requested data point.

Since it returns only a single data point, the output xarray dataset does not include a `time` dimension.

## Errors

<ParamField type="No such datapoint: <id>">
  The specified datapoint does not exist in this collection.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  data = collection.find(
      "0186d6b6-66cc-fcfd-91df-bbbff72499c3",
  )


  # check if a datapoint exists
  from tilebox.datasets.sync.dataset import NotFoundError

  try:
      collection.find(
          "0186d6b6-66cc-fcfd-91df-bbbff72499c3",
          skip_data=True,
      )
      exists = True
  except NotFoundError:
      exists = False
  ```
</RequestExample>


# Collection.info
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Collection.info



```python theme={"system"}
def Collection.info(
    availability: bool | None = None,
    count: bool | None = None,
) -> CollectionInfo
```

Fetch metadata about the data points in this collection.

Collection availability and datapoint counts are always returned. The `availability` and `count` arguments are deprecated and no longer affect the response.

## Parameters

<ParamField type="bool | None">
  Deprecated. Collection availability is always returned.
</ParamField>

<ParamField type="bool | None">
  Deprecated. Collection datapoint counts are always returned.
</ParamField>

## Returns

A collection info object.

<RequestExample>
  ```python Python theme={"system"}
  info = collection.info()
  ```
</RequestExample>


# Collection.ingest
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Collection.ingest



```python theme={"system"}
def Collection.ingest(
    data: IngestionData,
    allow_existing: bool = True,
    *,
    show_progress: bool | Callable[[float], None] = False,
) -> list[UUID]
```

Ingest data into a collection.

<Note>
  You need write permission on the collection to ingest data points.
</Note>

## Parameters

<ParamField type="IngestionData">
  The data to ingest.

  Supported `IngestionData` data types are:

  * A `pandas.DataFrame`, mapping the column names to dataset fields.
  * An `xarray.Dataset`, mapping variables and coordinates to dataset fields.
  * An `Iterable`, `dict` or `nd-array`: ingest any object that can be converted to a `pandas.DataFrame` using
    its constructor, equivalent to `ingest(pd.DataFrame(data))`.
</ParamField>

<ParamField type="bool">
  Datapoint fields are used to generate a deterministic unique `UUID` for each
  datapoint in a collection. Duplicate data points result in the same ID being generated.
  If `allow_existing` is `True`, `ingest` will skip those data points, since they already exist.
  If `allow_existing` is `False`, `ingest` will raise an error if any of the generated datapoint IDs already exist.
  Defaults to `True`.
</ParamField>

<ParamField type="bool | Callable[[float], None]">
  If `True`, display a progress bar while ingesting many datapoints. You can also pass a callback to receive progress values between `0` and `1`. Defaults to `False`.
</ParamField>

## Returns

List of datapoint IDs that were ingested, including the IDs of existing data points in case of duplicates and
`allow_existing=True`.

<RequestExample>
  ```python Python theme={"system"}
  import pandas as pd

  collection.ingest(pd.DataFrame({
      "time": [
        "2023-05-01T12:00:00Z",
        "2023-05-02T12:00:00Z",
      ],
      "value": [1, 2],
      "sensor": ["A", "B"],
  }))
  ```
</RequestExample>

## Errors

<ParamField type="found existing datapoints with same id">
  If `allow_existing` is `False` and any of the datapoints attempting to ingest already exist.
</ParamField>


# Collection.query
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Collection.query



```python theme={"system"}
def Collection.query(
    *,
    temporal_extent: TimeIntervalLike,
    spatial_extent: SpatialFilterLike | None = None,
    skip_data: bool = False,
    show_progress: bool | Callable[[float], None] = False,
) -> xarray.Dataset
```

Query a range of data points in this collection in a specified temporal extent.
If no data exists for the requested time or interval, an empty `xarray.Dataset` is returned.

## Parameters

<ParamField type="TimeIntervalLike">
  The time or time interval for which to query data. This can be a single time scalar, a tuple of two time scalars, or an array of time scalars.

  Valid time scalars are: `datetime.datetime` objects, strings in ISO 8601 format, or Unix timestamps in seconds.

  Behavior for each input type:

  * **TimeScalar**: If a single time scalar is provided, `query` returns all data points for that exact millisecond.

  * **TimeInterval**: If a time interval is provided, `query` returns all data points in that interval. Intervals can be a tuple of two `TimeScalars` or a `TimeInterval` object. Tuples are interpreted as a half-open interval `[start, end)`. With a `TimeInterval` object, the `start_exclusive` and `end_inclusive` parameters control whether the start and end time are inclusive or exclusive.

  * **Iterable\[TimeScalar]**: If an array of time scalars is provided, `query` constructs a time interval from the first and last time scalar in the array. Here, both the `start` and `end` times are inclusive.
</ParamField>

<ParamField type="SpatialFilterLike | None">
  Optional spatial filter. Use this for spatial queries in spatio-temporal datasets.
</ParamField>

<ParamField type="bool">
  If `True`, only required datapoint fields are returned, such as `time`, `id`, and `ingestion_time`. Defaults to `False`.
</ParamField>

<ParamField type="bool | Callable[[float], None]">
  If `True`, display a progress bar when pagination is required. You can also pass a callback to receive progress values between `0` and `1`. Defaults to `False`.
</ParamField>

## Returns

An [`xarray.Dataset`](/sdks/python/xarray) containing the requested data points.

<RequestExample>
  ```python Python theme={"system"}
  from datetime import datetime
  from tilebox.datasets.query import TimeInterval

  # querying a specific time
  time = "2023-05-01 12:45:33.423"
  data = collection.query(temporal_extent=time)

  # querying a time interval
  interval = ("2023-05-01", "2023-08-01")
  data = collection.query(temporal_extent=interval)

  # displaying a progress bar while querying
  data = collection.query(
    temporal_extent=interval,
    show_progress=True,
  )

  # querying a spatio-temporal collection with a spatial filter
  data = collection.query(
    temporal_extent=interval,
    spatial_extent=geometry,
  )

  # querying a time interval with TimeInterval
  interval = TimeInterval(
      start=datetime(2023, 5, 1),
      end=datetime(2023, 8, 1),
      start_exclusive=False,
      end_inclusive=False,
  )
  data = collection.query(temporal_extent=interval)

  # querying with an iterable
  datapoints = collection.query(
    temporal_extent=interval,
    skip_data=True,  # only fetch datapoint IDs and time
  )
  first_50 = collection.query(temporal_extent=datapoints.time[:50])
  ```
</RequestExample>


# Dataset.collection
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.collection



```python theme={"system"}
def Dataset.collection(name: str) -> Collection
```

Access a specific collection by its name.

## Parameters

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

A collection object.

<RequestExample>
  ```python Python theme={"system"}
  collection = dataset.collection("My-collection")
  ```
</RequestExample>

## Errors

<ParamField type="No such collection: Non-existent-Collection">
  The specified collection does not exist in the dataset.
</ParamField>


# Dataset.collections
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.collections



```python theme={"system"}
def Dataset.collections(
    availability: bool | None = None,
    count: bool | None = None,
) -> dict[str, Collection]
```

List the available collections in a dataset.

Collection availability and datapoint counts are always returned. The `availability` and `count` arguments are deprecated and no longer affect the response.

## Parameters

<ParamField type="bool | None">
  Deprecated. Collection availability is always returned.
</ParamField>

<ParamField type="bool | None">
  Deprecated. Collection datapoint counts are always returned.
</ParamField>

## Returns

A dictionary mapping collection names to collection objects.

<RequestExample>
  ```python Python theme={"system"}
  collections = dataset.collections()
  ```
</RequestExample>


# Dataset.create_collection
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.create_collection



```python theme={"system"}
def Dataset.create_collection(name: str) -> Collection
```

Create a collection in the dataset.

## Parameters

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

The created collection object.

<RequestExample>
  ```python Python theme={"system"}
  collection = dataset.create_collection("My-collection")
  ```
</RequestExample>


# Dataset.delete_collection
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.delete_collection



```python theme={"system"}
def Dataset.delete_collection(collection: str | UUID | CollectionClient) -> None
```

Delete a collection in the dataset.

## Parameters

<ParamField type="str | UUID | CollectionClient">
  The collection to delete. Can be specified by name, id, or as a collection object.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  dataset.delete_collection("My-collection")
  ```
</RequestExample>


# Dataset.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.find



```python theme={"system"}
def Dataset.find(
    datapoint_id: str | UUID,
    collections: list[str] | list[UUID] | list[Collection] | list[CollectionInfo] | list[CollectionClient] | None = None,
    skip_data: bool = False,
) -> xarray.Dataset
```

Find a specific datapoint by ID across one or more collections in this dataset.

If `collections` is not provided, all collections in the dataset are searched.

## Parameters

<ParamField type="str | UUID">
  The ID of the datapoint to find.
</ParamField>

<ParamField type="list[str | UUID | Collection | CollectionInfo | CollectionClient] | None">
  Optional collection scope for the lookup.

  Supported values include collection names, collection IDs, and collection objects. If omitted or set to `None`, all collections in the dataset are searched.
</ParamField>

<ParamField type="bool">
  If `True`, only required datapoint fields are returned (`time`, `id`, `ingestion_time`). Defaults to `False`.
</ParamField>

## Returns

An [`xarray.Dataset`](/sdks/python/xarray) containing the requested datapoint.

Since it returns only a single datapoint, the output dataset does not include a `time` dimension.

## Errors

<ParamField type="Invalid datapoint id: <id> is not a valid UUID">
  Raised when `datapoint_id` is not a valid UUID.
</ParamField>

<ParamField type="Collection <name> not found in dataset <dataset>">
  Raised when one or more collection names do not exist in the dataset.
</ParamField>

<ParamField type="Collection <id> is not part of the dataset <dataset>">
  Raised when one or more provided collection IDs/objects are not part of the dataset.
</ParamField>

<ParamField type="No such datapoint <id>">
  Raised when the datapoint cannot be found in the selected collections.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  # search all collections in the dataset
  datapoint = dataset.find("0186d6b6-66cc-fcfd-91df-bbbff72499c3")

  # search specific collections only
  datapoint = dataset.find(
      "0186d6b6-66cc-fcfd-91df-bbbff72499c3",
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A"],
  )

  # check if a datapoint exists
  from tilebox.datasets.sync.dataset import NotFoundError

  try:
      dataset.find(
          "0186d6b6-66cc-fcfd-91df-bbbff72499c3",
          collections=["S2A_S2MSI2A"],
          skip_data=True,
      )
      exists = True
  except NotFoundError:
      exists = False
  ```
</RequestExample>


# Dataset.get_or_create_collection
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.get_or_create_collection



```python theme={"system"}
def Dataset.get_or_create_collection(name: str) -> Collection
```

Get a collection by its name. If the collection does not exist, it will be created.

## Parameters

<ParamField type="string">
  The name of the collection
</ParamField>

## Returns

A collection object.

<RequestExample>
  ```python Python theme={"system"}
  collection = dataset.get_or_create_collection("My-collection")
  ```
</RequestExample>


# Dataset.query
Source: https://docs.tilebox.com/api-reference/python/tilebox.datasets/Dataset.query



```python theme={"system"}
def Dataset.query(
    *,
    collections: list[str] | list[UUID] | list[Collection] | list[CollectionInfo] | list[CollectionClient] | dict[str, CollectionClient] | None = None,
    temporal_extent: TimeIntervalLike,
    spatial_extent: SpatialFilterLike | None = None,
    skip_data: bool = False,
    show_progress: bool | Callable[[float], None] = False,
) -> xarray.Dataset
```

Query data points across one or more collections in this dataset.

If `collections` is not provided, all collections in the dataset are queried.
If no data matches the filters, an empty `xarray.Dataset` is returned.

## Parameters

<ParamField type="list[...] | dict[str, CollectionClient] | None">
  Optional collection scope for the query.

  Supported values include:

  * A list of collection names (`list[str]`)
  * A list of collection IDs (`list[UUID]`)
  * A list of collection objects (`list[Collection]`, `list[CollectionInfo]`, `list[CollectionClient]`)
  * The dictionary returned by `dataset.collections()`

  If omitted or set to `None`, all collections in the dataset are queried.
</ParamField>

<ParamField type="TimeIntervalLike">
  The time or time interval to query. This can be a single time scalar, a tuple of two time scalars, or a `TimeInterval` object.
</ParamField>

<ParamField type="SpatialFilterLike | None">
  Optional spatial filter. Use this for spatial queries in spatio-temporal datasets.
</ParamField>

<ParamField type="bool">
  If `True`, only required datapoint fields are returned (`time`, `id`, `ingestion_time`). Defaults to `False`.
</ParamField>

<ParamField type="bool | Callable[[float], None]">
  If `True`, display a progress bar when pagination is required. You can also pass a callback to receive progress values between `0` and `1`. Defaults to `False`.
</ParamField>

## Returns

An [`xarray.Dataset`](/sdks/python/xarray) containing matching datapoints.

## Errors

<ParamField type="A temporal_extent for your query must be specified">
  Raised when `temporal_extent` is not provided.
</ParamField>

<ParamField type="Collection <name> not found in dataset <dataset>">
  Raised when one or more collection names do not exist in the dataset.
</ParamField>

<ParamField type="Collection <id> is not part of the dataset <dataset>">
  Raised when one or more provided collection IDs/objects are not part of the dataset.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  # query all collections in the dataset
  data = dataset.query(
      temporal_extent=("2025-04-01", "2025-05-01"),
  )

  # query selected collections by name
  data = dataset.query(
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A"],
      temporal_extent=("2025-04-01", "2025-05-01"),
      show_progress=True,
  )

  # query selected collections by object
  collections = dataset.collections()
  data = dataset.query(
      collections=[collections["S2A_S2MSI2A"], collections["S2B_S2MSI2A"]],
      temporal_extent=("2025-04-01", "2025-05-01"),
      skip_data=True,
  )
  ```
</RequestExample>


# AutomationClient.all
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.all



```python theme={"system"}
def AutomationClient.all() -> list[AutomationPrototype]
```

List all registered automations.

## Returns

A list of automation prototypes.

<RequestExample>
  ```python Python theme={"system"}
  automations = client.automations().all()
  ```
</RequestExample>


# AutomationClient.create_cron_automation
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.create_cron_automation



```python theme={"system"}
def AutomationClient.create_cron_automation(
    name: str,
    task: CronTask,
    cron_schedules: str | list[str],
    cluster: ClusterSlugLike | None = None,
    max_retries: int = 0,
) -> AutomationPrototype
```

Create an automation that submits a task on one or more cron schedules.

## Parameters

<ParamField type="str">
  Name of the automation to create.
</ParamField>

<ParamField type="CronTask">
  Task to run when the automation is triggered.
</ParamField>

<ParamField type="str | list[str]">
  Cron schedule or schedules that trigger the automation.
</ParamField>

<ParamField type="ClusterSlugLike | None">
  Cluster to run the task on. If not provided, the default cluster is used.
</ParamField>

<ParamField type="int">
  Maximum number of retries for the task. Defaults to `0`.
</ParamField>

## Returns

The created automation prototype.

<RequestExample>
  ```python Python theme={"system"}
  automation = client.automations().create_cron_automation(
      name="daily-index-refresh",
      task=RefreshIndex(),
      cron_schedules="0 2 * * *",
      cluster="default",
  )
  ```
</RequestExample>


# AutomationClient.create_storage_event_automation
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.create_storage_event_automation



```python theme={"system"}
def AutomationClient.create_storage_event_automation(
    name: str,
    task: StorageEventTask,
    triggers: list[tuple[StorageLocation, str]] | tuple[StorageLocation, str],
    cluster: ClusterSlugLike | None = None,
    max_retries: int = 0,
) -> AutomationPrototype
```

Create an automation that submits a task when objects are added to storage.

## Parameters

<ParamField type="str">
  Name of the automation to create.
</ParamField>

<ParamField type="StorageEventTask">
  Task to run when a matching storage event occurs.
</ParamField>

<ParamField type="list[tuple[StorageLocation, str]] | tuple[StorageLocation, str]">
  One trigger or a list of triggers. Each trigger contains a storage location and a glob pattern.
</ParamField>

<ParamField type="ClusterSlugLike | None">
  Cluster to run the task on. If not provided, the default cluster is used.
</ParamField>

<ParamField type="int">
  Maximum number of retries for the task. Defaults to `0`.
</ParamField>

## Returns

The created automation prototype.

<RequestExample>
  ```python Python theme={"system"}
  location = client.automations().storage_locations()[0]

  automation = client.automations().create_storage_event_automation(
      name="ingest-new-scenes",
      task=IngestScene(),
      triggers=(location, "incoming/**/*.json"),
  )
  ```
</RequestExample>


# AutomationClient.delete
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.delete



```python theme={"system"}
def AutomationClient.delete(
    automation_or_id: AutomationPrototype | UUID | str,
    cancel_jobs: bool = False,
) -> None
```

Delete an automation by object or ID.

## Parameters

<ParamField type="AutomationPrototype | UUID | str">
  Automation object, automation ID, or automation ID string to delete.
</ParamField>

<ParamField type="bool">
  Whether to cancel currently queued or running jobs for the automation. Defaults to `False`.
</ParamField>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  client.automations().delete(automation, cancel_jobs=True)
  ```
</RequestExample>


# AutomationClient.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.find



```python theme={"system"}
def AutomationClient.find(automation_id: UUID | str) -> AutomationPrototype
```

Find an automation by ID.

## Parameters

<ParamField type="UUID | str">
  ID of the automation to fetch.
</ParamField>

## Returns

The automation prototype for the given ID.

<RequestExample>
  ```python Python theme={"system"}
  automation = client.automations().find("0195c87a-49f6-5ffa-e3cb-92215d057ea6")
  ```
</RequestExample>


# AutomationClient.storage_locations
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/AutomationClient.storage_locations



```python theme={"system"}
def AutomationClient.storage_locations() -> list[StorageLocation]
```

List storage locations that can be used as storage event automation triggers.

## Returns

A list of storage locations available to the current account.

<RequestExample>
  ```python Python theme={"system"}
  locations = client.automations().storage_locations()
  ```
</RequestExample>


# Client
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Client



```python theme={"system"}
class Client(
    *,
    url: str = "https://api.tilebox.com",
    token: str | None = None,
    name: str | None = None,
    client_id: UUID | None = None,
    transport: Literal["grpc", "http1"] = "grpc",
)
```

Create a Tilebox workflows client.

## Parameters

<ParamField type="str">
  Tilebox API Url. Defaults to `https://api.tilebox.com`.
</ParamField>

<ParamField type="str | None">
  The API key to authenticate with. If not set, the `TILEBOX_API_KEY` environment variable is used.
</ParamField>

<ParamField type="str | None">
  Optional service name for workflow telemetry. If not set, the default service name is used.
</ParamField>

<ParamField type="UUID | None">
  Optional stable ID used to scope internal loggers. If not set, a random ID is generated.
</ParamField>

<ParamField type="Literal[&#x22;grpc&#x22;, &#x22;http1&#x22;]">
  Network transport to use for API requests. Defaults to `"grpc"`. Use `"http1"` to force the Connect protocol over HTTP/1.1 on networks that do not support gRPC over HTTP/2 correctly.
</ParamField>

## Sub clients

The workflows client exposes sub clients for interacting with different parts of the Tilebox workflows API.

```python theme={"system"}
def Client.jobs() -> JobClient
```

A client for interacting with jobs.

```python theme={"system"}
def Client.clusters() -> ClusterClient
```

A client for managing clusters.

```python theme={"system"}
def Client.workflows() -> WorkflowClient
```

A client for managing workflows and workflow release deployments.

```python theme={"system"}
def Client.automations() -> AutomationClient
```

A client for scheduling automations.

## Logging

```python theme={"system"}
def Client.configure_logging(
    level: int | logging.Logger,
    runner_level: int | None = None,
) -> None
```

Configure which task and runner logs this client exports. See [`Client.configure_logging`](/api-reference/python/tilebox.workflows/Client.configure_logging).

## Runners

```python theme={"system"}
def Client.runner(...) -> TaskRunner
```

A client is also used to instantiate runners. Check out the [`Client.runner` API reference](/api-reference/python/tilebox.workflows/Client.runner) for more information.

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  # read token from an environment variable
  client = Client()

  # or provide connection details directly
  client = Client(
      url="https://api.tilebox.com",
      token="YOUR_TILEBOX_API_KEY",
      name="sentinel-2-runner",
  )

  # access sub clients
  job_client = client.jobs()
  cluster_client = client.clusters()
  workflow_client = client.workflows()
  automation_client = client.automations()

  # or instantiate a runner
  runner = client.runner(tasks=[...])
  ```
</RequestExample>


# Client.configure_logging
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Client.configure_logging



```python theme={"system"}
def Client.configure_logging(
    level: int | logging.Logger,
    runner_level: int | None = None,
) -> None
```

Configure the log level for logs exported by this workflow client.

## Parameters

<ParamField type="int | logging.Logger">
  Logging level for task logs emitted with `context.logger`, for example `logging.INFO` or `logging.DEBUG`.
</ParamField>

<ParamField type="int | None">
  Logging level for internal runner logs. If omitted, the value of `level` is used.
</ParamField>

<Note>
  Passing a `logging.Logger` instance as `level` is deprecated. Configure the Tilebox root logger directly when you need to export logs to another destination.
</Note>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  import logging

  from tilebox.workflows import Client

  client = Client(name="sentinel-2-runner")
  client.configure_logging(level=logging.DEBUG, runner_level=logging.INFO)
  ```
</RequestExample>


# Client.runner
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Client.runner



```python theme={"system"}
def Client.runner(
    cluster: ClusterSlugLike | None = None,
    tasks: list[type[Task]] | None = None,
    cache: JobCache | None = None,
    context: type[RunnerContext] | None = None,
    runner: Runner | None = None,
) -> TaskRunner
```

Initialize a direct runner from task classes.

For new Python workflow projects, define a reusable [`Runner`](/api-reference/python/tilebox.workflows/Runner) object and call [`Runner.connect_to`](/api-reference/python/tilebox.workflows/Runner.connect_to) when you want direct execution. The `Client.runner` method remains available for existing code and simple direct runner scripts.

## Parameters

<ParamField type="str | None">
  The [cluster slug](/workflows/concepts/clusters#cluster-slug) for the cluster associated with this direct runner.
  If not provided, the default cluster is used.
</ParamField>

<ParamField type="list[type[Task]] | None">
  A list of task classes that this runner can execute. Pass either `tasks`, `cache`, and `context`, or pass a reusable `runner` object.
</ParamField>

<ParamField type="JobCache">
  An optional [job cache](/workflows/run-and-inspect/caches) for caching results from tasks and sharing data between tasks.
</ParamField>

<ParamField type="type[RunnerContext] | None">
  Optional runner context class to instantiate for task execution.
</ParamField>

<ParamField type="Runner | None">
  A reusable runner definition. If provided, do not also pass `tasks`, `cache`, or `context`.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import LocalFileSystemCache

  client = Client()
  runner = client.runner(
      tasks=[MyFirstTask, MySubtask],
      # optional:
      cache=LocalFileSystemCache("cache_directory"),
  )
  ```
</RequestExample>


# ClusterClient.all
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ClusterClient.all



```python theme={"system"}
def ClusterClient.all(): list[Cluster]
```

List all available clusters.

## Returns

A list of all available clusters. Each cluster includes its slug, display name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```python Python theme={"system"}
  clusters = cluster_client.all()
  ```
</RequestExample>


# ClusterClient.create
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ClusterClient.create



```python theme={"system"}
def ClusterClient.create(
    name: str,
    description: str | None = None,
    slug: str | None = None,
) -> Cluster
```

Create a cluster.

## Parameters

<ParamField type="string">
  A display name for the cluster.
</ParamField>

<ParamField type="str | None">
  Optional cluster description.
</ParamField>

<ParamField type="str | None">
  Optional cluster slug. If omitted, Tilebox generates a slug from the cluster name.
</ParamField>

## Returns

The created cluster object, including its slug, display name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```python Python theme={"system"}
  cluster = cluster_client.create(
      "My cluster",
      description="Development workloads",
      slug="dev",
  )
  ```
</RequestExample>


# ClusterClient.delete
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ClusterClient.delete



```python theme={"system"}
def ClusterClient.delete(cluster_or_slug: Cluster | str)
```

Delete a cluster.

## Parameters

<ParamField type="Cluster | str">
  The cluster or cluster slug to delete.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  cluster_client.delete(cluster)
  ```
</RequestExample>


# ClusterClient.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ClusterClient.find



```python theme={"system"}
def ClusterClient.find(cluster_or_slug: Cluster | str): Cluster
```

Get a cluster by its slug.

## Parameters

<ParamField type="Cluster | str">
  The cluster or cluster slug to get.
</ParamField>

## Returns

A cluster object, including its slug, display name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```python Python theme={"system"}
  cluster = cluster_client.find("my-cluster-CvufcSxcC9SKfe")
  ```
</RequestExample>


# ClusterClient.update
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ClusterClient.update



```python theme={"system"}
def ClusterClient.update(
    cluster_or_slug: Cluster | str,
    name: str | None = None,
    description: str | None = None,
) -> Cluster
```

Update a cluster.

## Parameters

<ParamField type="Cluster | str">
  The cluster or cluster slug to update.
</ParamField>

<ParamField type="str | None">
  Optional new display name for the cluster. If omitted, the display name is unchanged.
</ParamField>

<ParamField type="str | None">
  Optional new cluster description. Pass an empty string to clear the description. If omitted, the description is unchanged.
</ParamField>

## Returns

The updated cluster object, including its slug, display name, description, whether it can be deleted, and deployed workflows.

<RequestExample>
  ```python Python theme={"system"}
  cluster = cluster_client.update(
      "dev",
      name="Development",
      description="Development workloads",
  )
  ```
</RequestExample>


# ExecutionContext.job_cache
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.job_cache



```python theme={"system"}
ExecutionContext.job_cache: JobCache
```

Access the job cache for a task.

<RequestExample>
  ```python Python theme={"system"}
  # within a Tasks execute method, access the job cache
  # def execute(self, context: ExecutionContext):
  context.job_cache["some-key"] = b"my-value"
  ```
</RequestExample>


# Context.logger
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.logger



```python theme={"system"}
ExecutionContext.logger: StructuredLogger
```

Structured logger for logs emitted by the currently executing task.

## Methods

```python theme={"system"}
context.logger.debug(message, /, *args, **attributes) -> None
context.logger.info(message, /, *args, **attributes) -> None
context.logger.warning(message, /, *args, **attributes) -> None
context.logger.error(message, /, *args, **attributes) -> None
context.logger.exception(message, /, *args, **attributes) -> None
context.logger.critical(message, /, *args, **attributes) -> None
context.logger.bind(**attributes) -> StructuredLogger
```

Structured attributes are attached to the log record and become searchable telemetry attributes.

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext, Task

  class DownloadInput(Task):
      input_id: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Started", input_id=self.input_id)

          log = context.logger.bind(component="download")
          log.debug("Fetching input")
  ```
</RequestExample>


# ExecutionContext.progress
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.progress



```python theme={"system"}
def ExecutionContext.progress(label: str | None = None) -> ProgressUpdate
```

Create or return a progress indicator for the currently executing task.

Progress updates are attached to the task result and are visible in job execution state. Calling `progress` again with the same label returns the same progress indicator.

## Parameters

<ParamField type="str | None">
  Optional label for the progress indicator. Empty strings are treated as `None`.
</ParamField>

## Returns

A `ProgressUpdate` object for tracking total and completed work.

<RequestExample>
  ```python Python theme={"system"}
  def execute(self, context: ExecutionContext) -> None:
      progress = context.progress("download-scenes")
      progress.add(len(self.scenes))

      for scene in self.scenes:
          download(scene)
          progress.done(1)
  ```
</RequestExample>


# ExecutionContext.runner_context
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.runner_context



```python theme={"system"}
ExecutionContext.runner_context: RunnerContext
```

Access the runner context instance for the runner executing the task.

Use a custom `RunnerContext` subclass when tasks need shared runtime configuration or helpers during execution.

## Returns

The runner context object associated with the runner.

<RequestExample>
  ```python Python theme={"system"}
  def execute(self, context: ExecutionContext) -> None:
      bucket = context.runner_context.gcs_client("my-project:my-bucket")
      blob = bucket.blob("inputs/scene.json")
      data = blob.download_as_bytes()
  ```
</RequestExample>


# Context.submit_subtask
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask



```python theme={"system"}
def ExecutionContext.submit_subtask(
  task: Task,
  depends_on: FutureTask | list[FutureTask] | None = None,
  cluster: str | None = None,
  max_retries: int = 0,
  optional: bool = False
) -> FutureTask
```

Submit a [subtask](/workflows/concepts/tasks#subtasks-and-task-composition) from a currently executing task.

## Parameters

<ParamField type="Task">
  The task to submit as a subtask.
</ParamField>

<ParamField type="FutureTask | list[FutureTask] | None">
  An optional task or list of tasks already submitted within the same context that this subtask depends on.
</ParamField>

<ParamField type="str">
  An optional [cluster slug](/workflows/concepts/clusters#managing-clusters) for running the subtask. If not provided, the subtask runs on the same cluster as the parent task.
</ParamField>

<ParamField type="int">
  Specify the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtask in case of failure. The default is 0.
</ParamField>

<ParamField type="bool">
  Whether the subtask is [optional](/workflows/concepts/tasks#optional-tasks). If `True`, the subtask will not fail the job if it fails. Tasks that depend on this task will still execute even if this task failed. The default is `False`.
</ParamField>

## Returns

A `FutureTask` object that represents the submitted subtask. Can be used to set up dependencies between tasks.

<RequestExample>
  ```python Python theme={"system"}
  # within the execute method of a Task:
  subtask = context.submit_subtask(MySubtask())
  dependent_subtask = context.submit_subtask(
      MyOtherSubtask(), depends_on=[subtask]
  )
  gpu_task = context.submit_subtask(
      MyGPUTask(),
      cluster="gpu-cluster-slug"
  )
  flaky_task = context.submit_subtask(
      MyFlakyTask(),
      max_retries=5
  )
  optional_task = context.submit_subtask(
      MyOptionalTask(),
      optional=True
  )
  ```
</RequestExample>


# Context.submit_subtasks
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtasks



```python theme={"system"}
def ExecutionContext.submit_subtasks(
  tasks: Sequence[Task],
  depends_on: FutureTask | list[FutureTask] | None = None,
  cluster: str | None = None,
  max_retries: int = 0,
  optional: bool = False
) -> list[FutureTask]
```

Submit multiple [subtasks](/workflows/concepts/tasks#subtasks-and-task-composition) from a currently executing task. Same as [submit\_subtask](/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask), but accepts a sequence of tasks.

## Parameters

<ParamField type="Sequence[Task]">
  The tasks to submit as subtasks.
</ParamField>

<ParamField type="FutureTask | list[FutureTask] | None">
  An optional task or list of tasks already submitted within the same context that the subtasks depend on.
</ParamField>

<ParamField type="str">
  An optional [cluster slug](/workflows/concepts/clusters#managing-clusters) for running the subtasks. If not provided, the subtasks run on the same cluster as the parent task.
</ParamField>

<ParamField type="int">
  Specify the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtasks in case of failure. The default is 0.
</ParamField>

<ParamField type="bool">
  Whether the subtasks are [optional](/workflows/concepts/tasks#optional-tasks). If `True`, the subtasks will not fail the job if they fail. Tasks that depend on them will still execute even if they failed. The default is `False`.
</ParamField>

## Returns

A list of `FutureTask` objects representing the submitted subtasks. Can be used to set up dependencies between tasks.

<RequestExample>
  ```python Python theme={"system"}
  # within the execute method of a Task:
  subtasks = context.submit_subtasks([
      MySubtask(i) for i in range(10)
  ])
  dependent_subtask = context.submit_subtask(
      MyOtherSubtask(), depends_on=subtasks
  )
  ```
</RequestExample>


# Context.tracer
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ExecutionContext.tracer



```python theme={"system"}
ExecutionContext.tracer: WorkflowTracer
```

Tracer for creating custom spans inside the currently executing task.

## Methods

```python theme={"system"}
context.tracer.span(name: str, *args, **kwargs) -> ContextManager[Span]
context.tracer.current_span() -> Span
```

Custom spans are nested under the current task span and are exported to Tilebox with the job trace.

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext, Task

  class DownloadInput(Task):
      input_id: str

      def execute(self, context: ExecutionContext) -> None:
          with context.tracer.span("download-input") as span:
              span.set_attribute("input_id", self.input_id)
  ```
</RequestExample>


# JobCache.__iter__
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobCache.__iter__



```python theme={"system"}
def JobCache.__iter__() -> Iterator[str]
```

List all available keys in a job cache group. Only keys that are direct children of the current group are returned.
For nested groups, first access the group and then iterate over its keys.

<RequestExample>
  ```python Python theme={"system"}
  # within a Tasks execute method, access the job cache
  # def execute(context: ExecutionContext)
  cache = context.job_cache

  # iterate over all keys
  for key in cache:
    print(cache[key])

  # or collect as list
  keys = list(cache)

  # also works with nested groups
  for key in cache.group("some-group"):
    print(cache[key])
  ```
</RequestExample>


# JobCache.group
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobCache.group



```python theme={"system"}
def JobCache.group(name: str) -> JobCache
```

You can nest caches in a hierarchical manner using [groups](/workflows/run-and-inspect/caches#groups-and-hierarchical-keys).
Groups are separated by a forward slash (/) in the key. This hierarchical structure functions similarly to a file system.

## Parameters

<ParamField type="str">
  The name of the cache group.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  # within a Tasks execute method, access the job cache
  # def execute(context: ExecutionContext)
  cache = context.job_cache

  # use groups to nest related cache values
  group = cache.group("some-group")
  group["value"] = b"my-value"

  nested = group.group("nested")
  nested["value"] = b"nested-value"

  # access nested groups directly via / notation
  value = cache["some-group/nested/value"]
  ```
</RequestExample>


# JobClient.cancel
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.cancel



```python theme={"system"}
def JobClient.cancel(job_or_id: Job | str)
```

Cancel a job. When a job is canceled, no queued tasks will be picked up by runners and executed even if runners are idle. Tasks that are already being executed will finish their execution and not be interrupted. All sub-tasks spawned from such tasks after the cancellation will not be picked up by runners.

## Parameters

<ParamField type="Job | str">
  The job or job id to cancel.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  job_client.cancel(job)
  ```
</RequestExample>


# JobClient.display
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.display



```python theme={"system"}
def JobClient.display(
    job_id: Job | UUID | str,
    direction: str = "down",
    layout: str = "dagre",
    sketchy: bool = True,
) -> None
```

Display a job diagram in an interactive Python environment, such as a Jupyter notebook.

Use [`JobClient.visualize`](/api-reference/python/tilebox.workflows/JobClient.visualize) when you need the SVG markup instead of displaying it directly.

## Parameters

<ParamField type="Job | UUID | str">
  The job, job ID, or job ID string to display.
</ParamField>

<ParamField type="str">
  Direction for the diagram to flow. Defaults to `down`.
</ParamField>

<ParamField type="str">
  Layout engine for the diagram. Supported values are `dagre` and `elk`. Defaults to `dagre`.
</ParamField>

<ParamField type="bool">
  Whether to render the diagram in a sketchy, hand-drawn style. Defaults to `True`.
</ParamField>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  job_client.display(job)
  ```
</RequestExample>


# JobClient.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.find



```python theme={"system"}
def JobClient.find(job_or_id: Job | str) -> Job
```

Get a job by its id. Can also be an existing job object, in which case this method acts as a refresh operation to fetch the latest job details.

## Parameters

<ParamField type="Job | str">
  The job or job id to get.
</ParamField>

## Returns

A job object.

<RequestExample>
  ```python Python theme={"system"}
  job = job_client.find("0195c87a-49f6-5ffa-e3cb-92215d057ea6")
  ```
</RequestExample>


# JobClient.query
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.query



```python theme={"system"}
def JobClient.query(
    temporal_extent: TimeIntervalLike | IDIntervalLike,
    automation_ids: UUID | list[UUID] | None = None,
    job_states: JobState | list[JobState] | None = None,
    name: str | None = None,
    task_states: TaskState | list[TaskState] | None = None,
    clusters: Cluster | str | list[Cluster | str] | None = None,
) -> list[Job]
```

Query jobs in the specified interval.

## Parameters

<ParamField type="TimeIntervalLike | IDIntervalLike">
  The temporal extent to filter jobs by. If an `IDInterval` is given, jobs are filtered by their job id instead of their creation time.
  It can be specified in the following ways:

  * TimeInterval: interval -> Use the time interval as its given
  * DatetimeScalar: \[time, time] -> Construct a TimeInterval with start and end time set to the given
    value and the end time inclusive
  * tuple of two DatetimeScalar: \[start, end) -> Construct a TimeInterval with the given start and
    end time
  * `IDInterval`: interval -> Use the ID interval as its given
  * tuple of two UUIDs: \[start, end) -> Construct an `IDInterval` with the given start and end id
  * tuple of two strings: \[start, end) -> Construct an `IDInterval` with the given start and end id
    parsed from the strings
</ParamField>

<ParamField type="UUID | list[UUID] | None">
  One automation ID or a list of automation IDs to filter jobs by. If not provided, jobs from all automations are returned.
</ParamField>

<ParamField type="JobState | list[JobState] | None">
  A job state or list of job states to filter by. If specified, only jobs in any of the given states are returned.
</ParamField>

<ParamField type="str | None">
  A name to filter jobs by. If specified, only jobs with a matching name are returned. The name is matched using case-insensitive fuzzy search.
</ParamField>

<ParamField type="TaskState | list[TaskState] | None">
  A task state or list of task states to filter jobs by. If specified, only jobs that have at least one task in any of the given states are returned. Useful for finding jobs with [optional](/workflows/concepts/tasks#optional-tasks) task failures, e.g. `task_states=TaskState.FAILED_OPTIONAL`.
</ParamField>

<ParamField type="Cluster | str | list[Cluster | str] | None">
  A cluster or list of clusters to filter jobs by. Pass actual cluster slugs or cluster objects. `None` and `[]` apply no cluster filter. Empty string cluster slugs are not accepted.
</ParamField>

## Returns

A list of jobs.

<RequestExample>
  ```python Python theme={"system"}
  jobs = job_client.query(("2025-01-01", "2025-02-01"))
  jobs_on_cluster = job_client.query(("2025-01-01", "2025-02-01"), clusters="production")
  ```
</RequestExample>


# JobClient.query_logs
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.query_logs



```python theme={"system"}
def JobClient.query_logs(job_id: Job | UUID | str) -> LogRecords
```

Query logs emitted while running a job. Pagination is handled automatically.

## Parameters

<ParamField type="Job | UUID | str">
  The job, job ID, or job ID string to query logs for.
</ParamField>

## Returns

A `LogRecords` list. Each `LogRecord` contains `time`, `severity_number`, `severity_text`, `body`, `trace_id`, `span_id`, `attributes`, and `runner_attributes`.

`LogRecords.to_pandas()` converts the records to a pandas DataFrame.

<RequestExample>
  ```python Python theme={"system"}
  logs = client.jobs().query_logs(job)

  for record in logs:
      print(record.time, record.severity_text, record.body)

  df = logs.to_pandas()
  ```
</RequestExample>


# JobClient.query_spans
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.query_spans



```python theme={"system"}
def JobClient.query_spans(job_id: Job | UUID | str) -> Spans
```

Query spans emitted while running a job. Pagination is handled automatically.

## Parameters

<ParamField type="Job | UUID | str">
  The job, job ID, or job ID string to query spans for.
</ParamField>

## Returns

A `Spans` list. Each `Span` contains `start_time`, `end_time`, `duration`, `trace_id`, `span_id`, `parent_span_id`, `name`, `status_code`, `status_message`, `attributes`, `runner_attributes`, and `events`.

`Spans.to_pandas()` converts the spans to a pandas DataFrame and includes a computed `duration` column.

<RequestExample>
  ```python Python theme={"system"}
  spans = client.jobs().query_spans(job.id)

  for span in spans:
      print(span.name, span.status_code, span.duration)

  df = spans.to_pandas()
  ```
</RequestExample>


# JobClient.retry
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.retry



```python theme={"system"}
def JobClient.retry(job_or_id: Job | str) -> int
```

Retry a job. All failed tasks will become queued again, and queued tasks will be picked up by runners again.

## Parameters

<ParamField type="Job | str">
  The job or job id to retry.
</ParamField>

## Returns

The number of tasks that were rescheduled.

<RequestExample>
  ```python Python theme={"system"}
  nb_rescheduled = job_client.retry(job)
  ```
</RequestExample>


# JobClient.submit
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.submit



```python theme={"system"}
def JobClient.submit(
  job_name: str,
  root_task_or_tasks: Task | list[Task],
  cluster: ClusterSlugLike | list[ClusterSlugLike] | None = None,
  max_retries: int = 0
) -> Job
```

Submit a job.

## Parameters

<ParamField type="str">
  The name of the job.
</ParamField>

<ParamField type="Task | list[Task]">
  The root task or root tasks for the job. Root tasks execute first and can submit subtasks to compose the workflow.
</ParamField>

<ParamField type="ClusterSlugLike | list[ClusterSlugLike] | None">
  The [cluster slug](/workflows/concepts/clusters#managing-clusters) or cluster object for the root task. When submitting multiple root tasks, pass one cluster or a list with one cluster per task. If not provided, the default cluster is used.
</ParamField>

<ParamField type="int">
  The maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the root task or tasks. Defaults to 0.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  from my_workflow import MyTask

  job = job_client.submit(
      "my-job",
      MyTask(
        message="Hello, World!", 
        value=42,
        data={"key": "value"}
      ),
  )
  ```
</RequestExample>


# JobClient.visualize
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/JobClient.visualize



```python theme={"system"}
def JobClient.visualize(
  job: Job | UUID | str,
  direction: str = "down",
  layout: str = "dagre",
  sketchy: bool = True
) -> str
```

Create a visualization of a job as a diagram.

<Tip>
  If you are working in an interactive environment, such as Jupyter notebooks, you can use the `display` method to display the visualized job diagram directly in the notebook.
</Tip>

## Parameters

<ParamField type="Job | UUID | str">
  The job, job ID, or job ID string to visualize.
</ParamField>

<ParamField type="str">
  The direction for the diagram to flow. For more details, see the relevant section in the [D2 documentation](https://d2lang.com/tour/layouts/#direction). Valid values are `up`, `down`, `right`, and `left`. The default value is `down`.
</ParamField>

<ParamField type="str">
  The layout engine for the diagram. For further information, refer to the [D2 layout engines](https://d2lang.com/tour/layouts). Valid values are `dagre` and `elk`, with the default being `dagre`.
</ParamField>

<ParamField type="bool">
  Indicates whether to use a sketchy, hand-drawn style for the diagram. The default is `True`.
</ParamField>

## Returns

Rendered SVG markup for the job diagram.

<RequestExample>
  ```python Python theme={"system"}
  svg = job_client.visualize(job)
  Path("diagram.svg").write_text(svg)

  # in a notebook
  job_client.display(job)
  ```
</RequestExample>


# ProgressUpdate.add
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ProgressUpdate.add



```python theme={"system"}
def ProgressUpdate.add(count: int) -> None
```

Add work to the total amount tracked by a progress indicator.

Call `add` before or during a task when the task discovers more work to complete.

## Parameters

<ParamField type="int">
  Amount of work to add to the progress indicator total.
</ParamField>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  progress = context.progress("process-items")
  progress.add(len(items))
  ```
</RequestExample>


# ProgressUpdate.done
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/ProgressUpdate.done



```python theme={"system"}
def ProgressUpdate.done(count: int) -> None
```

Mark work as completed on a progress indicator.

Call `done` as the task finishes units of work added with [`ProgressUpdate.add`](/api-reference/python/tilebox.workflows/ProgressUpdate.add).

## Parameters

<ParamField type="int">
  Amount of work to mark as completed.
</ParamField>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  progress = context.progress("process-items")
  progress.add(len(items))

  for item in items:
      process(item)
      progress.done(1)
  ```
</RequestExample>


# Runner
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Runner



```python theme={"system"}
class Runner(
    *,
    tasks: list[type[Task]] | None = None,
    cache: JobCache | None = None,
    context: type[RunnerContext] | None = None,
)
```

Define task registrations for a Python workflow project.

A `Runner` object is a reusable definition. Use it for direct execution by connecting it to a [`Client`](/api-reference/python/tilebox.workflows/Client), or reference it from `tilebox.workflow.toml` so release runners can load the same task registrations from a workflow release.

## Parameters

<ParamField type="list[type[Task]] | None">
  A list of task classes this runner can execute.
</ParamField>

<ParamField type="JobCache | None">
  Optional [job cache](/workflows/run-and-inspect/caches) used by tasks executed by this runner.
</ParamField>

<ParamField type="type[RunnerContext] | None">
  Optional runner context class to instantiate for tasks executed by this runner.
</ParamField>

## Example

```python Python theme={"system"}
from tilebox.workflows import Runner
from tilebox.workflows.cache import LocalFileSystemCache

from my_workflow.tasks import MyRootTask, MySubtask


runner = Runner(
    tasks=[MyRootTask, MySubtask],
    cache=LocalFileSystemCache(),
)
```

You can also register tasks after creating a runner:

```python Python theme={"system"}
runner = Runner(cache=LocalFileSystemCache())
runner.register(MyRootTask)
runner.register(MySubtask)
```

Use the same object for direct execution:

```python Python theme={"system"}
from tilebox.workflows import Client

from my_workflow.runner import runner


runner.connect_to(Client(), cluster="dev-cluster").run_forever()
```

Or reference it from `tilebox.workflow.toml` for release execution:

```toml theme={"system"}
[workflow]
slug = "my-workflow"
root = "."
runner = "my_workflow.runner:runner"
```


# Runner.connect_to
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Runner.connect_to



```python theme={"system"}
def Runner.connect_to(
  client: Client,
  cluster: ClusterSlugLike | None = None,
) -> TaskRunner
```

Create a direct runner from a reusable [`Runner`](/api-reference/python/tilebox.workflows/Runner) definition and a Tilebox workflows client.

The returned [`TaskRunner`](/api-reference/python/tilebox.workflows/TaskRunner.run_forever) connects to the Tilebox API, advertises the task registrations from the `Runner` definition, and executes matching tasks from the selected cluster.

## Parameters

<ParamField type="Client">
  Tilebox workflows client used by the direct runner.
</ParamField>

<ParamField type="str | None">
  The [cluster slug](/workflows/concepts/clusters#cluster-slug) for the runner. If not provided, the default cluster is used.
</ParamField>

## Example

```python Python theme={"system"}
from tilebox.workflows import Client

from my_workflow.runner import runner


client = Client(name="my-workflow-direct")
task_runner = runner.connect_to(client, cluster="dev-cluster")
task_runner.run_forever()
```


# Runner.register
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Runner.register



```python theme={"system"}
def Runner.register(task: type[Task]) -> None
```

Register a task class with a reusable runner definition.

Use `register` when you want to build a runner incrementally instead of passing all task classes to the [`Runner`](/api-reference/python/tilebox.workflows/Runner) constructor.

## Parameters

<ParamField type="type[Task]">
  Task class that this runner can execute.
</ParamField>

## Returns

`None`

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import Runner

  from my_workflow.tasks import MyRootTask, MySubtask

  runner = Runner()
  runner.register(MyRootTask)
  runner.register(MySubtask)
  ```
</RequestExample>


# Task
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/Task



```python theme={"system"}
class Task:
    def execute(context: ExecutionContext) -> None

    @staticmethod
    def identifier() -> tuple[str, str]
```

Base class for Tilebox workflows [tasks](/workflows/concepts/tasks). Inherit from this class to create a task.
Inheriting also automatically applies the dataclass decorator.

## Methods

```python theme={"system"}
def Task.execute(context: ExecutionContext) -> None
```

The entry point for the execution of the task.

```python theme={"system"}
@staticmethod
def Task.identifier() -> tuple[str, str]
```

Override a task identifier and specify its version. If not overridden, the identifier of a task defaults to the class name, and the version to `v0.0`.

## Task Input Parameters

```python theme={"system"}
class MyTask(Task):
    message: str
    value: int
    data: dict[str, int]
```

Optional task [input parameters](/workflows/concepts/tasks#input-parameters), defined as class attributes. Supported types
are `str`, `int`, `float`, `bool`, as well as `lists` and `dicts` thereof.

<RequestExample>
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class MyFirstTask(Task):
      def execute(self, context: ExecutionContext):
          print(f"Hello World!")

      @staticmethod
      def identifier() -> tuple[str, str]:
          return ("tilebox.workflows.MyTask", "v3.2")

  class MyFirstParameterizedTask(Task):
      name: str
      greet: bool
      data: dict[str, str]

      def execute(self, context: ExecutionContext):
          if self.greet:
              print(f"Hello {self.name}!")
  ```
</RequestExample>


# TaskRunner.run_all
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/TaskRunner.run_all



```python theme={"system"}
def TaskRunner.run_all()
```

Run the runner and execute all tasks, until there are no more tasks available.

<RequestExample>
  ```python Python theme={"system"}
  # run all tasks until no more tasks are available
  runner.run_all()
  ```
</RequestExample>


# TaskRunner.run_forever
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/TaskRunner.run_forever



```python theme={"system"}
def TaskRunner.run_forever()
```

Run the runner forever. This will poll for new tasks and execute them as they come in.
If no tasks are available, it will sleep for a short time and then try again.

<RequestExample>
  ```python Python theme={"system"}
  # run forever, until the current process is stopped
  runner.run_forever()
  ```
</RequestExample>


# WorkflowClient.all
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.all



```python theme={"system"}
def WorkflowClient.all() -> list[Workflow]
```

List all workflows.

## Returns

A list of workflow objects.

<RequestExample>
  ```python Python theme={"system"}
  workflows = workflow_client.all()
  ```
</RequestExample>


# WorkflowClient.create
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.create



```python theme={"system"}
def WorkflowClient.create(name: str, description: str = "") -> Workflow
```

Create a workflow.

## Parameters

<ParamField type="str">
  The workflow display name.
</ParamField>

<ParamField type="str">
  Optional workflow description.
</ParamField>

## Returns

The created workflow object.

<RequestExample>
  ```python Python theme={"system"}
  workflow = workflow_client.create(
      "Scene processing",
      description="Process new scenes into analysis-ready outputs.",
  )
  ```
</RequestExample>


# WorkflowClient.delete
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.delete



```python theme={"system"}
def WorkflowClient.delete(workflow_or_slug: Workflow | str) -> None
```

Delete a workflow.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug to delete.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  workflow_client.delete("scene-processing")
  ```
</RequestExample>


# WorkflowClient.deploy_release
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.deploy_release



```python theme={"system"}
def WorkflowClient.deploy_release(
    workflow_or_slug: Workflow | str,
    release_or_id: WorkflowRelease | UUID | str,
    clusters: Cluster | str | list[Cluster | str] | None = None,
) -> WorkflowReleaseDeployment
```

Deploy a workflow release to one or more clusters.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug containing the release to deploy.
</ParamField>

<ParamField type="WorkflowRelease | UUID | str">
  The workflow release or release ID to deploy.
</ParamField>

<ParamField type="Cluster | str | list[Cluster | str] | None">
  The cluster or clusters to deploy the release to. If omitted, the API uses the default cluster.
</ParamField>

## Returns

The deployment result, including the release and affected clusters.

<RequestExample>
  ```python Python theme={"system"}
  deployment = workflow_client.deploy_release(
      workflow,
      release.id,
      clusters="production",
  )
  ```
</RequestExample>


# WorkflowClient.find
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.find



```python theme={"system"}
def WorkflowClient.find(workflow_or_slug: Workflow | str) -> Workflow
```

Get a workflow by slug.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug to get.
</ParamField>

## Returns

A workflow object.

<RequestExample>
  ```python Python theme={"system"}
  workflow = workflow_client.find("scene-processing")
  ```
</RequestExample>


# WorkflowClient.undeploy_release
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.undeploy_release



```python theme={"system"}
def WorkflowClient.undeploy_release(
    workflow_or_slug: Workflow | str,
    release_or_id: WorkflowRelease | UUID | str,
    clusters: Cluster | str | list[Cluster | str] | None = None,
) -> WorkflowReleaseDeployment
```

Remove a workflow release from one or more clusters.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug containing the release to remove from clusters.
</ParamField>

<ParamField type="WorkflowRelease | UUID | str">
  The workflow release or release ID to remove from clusters.
</ParamField>

<ParamField type="Cluster | str | list[Cluster | str] | None">
  The cluster or clusters to remove the release from. If omitted, the API uses the default cluster.
</ParamField>

## Returns

The result, including the release and affected clusters.

<RequestExample>
  ```python Python theme={"system"}
  deployment = workflow_client.undeploy_release(
      workflow,
      release.id,
      clusters="production",
  )
  ```
</RequestExample>


# WorkflowClient.unpublish_release
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.unpublish_release



```python theme={"system"}
def WorkflowClient.unpublish_release(
    workflow_or_slug: Workflow | str,
    release_or_id: WorkflowRelease | UUID | str,
) -> None
```

Remove a workflow release from the set of published releases.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug containing the release to remove.
</ParamField>

<ParamField type="WorkflowRelease | UUID | str">
  The workflow release or release ID to remove.
</ParamField>

<RequestExample>
  ```python Python theme={"system"}
  workflow_client.unpublish_release(workflow, release.id)
  ```
</RequestExample>


# WorkflowClient.update
Source: https://docs.tilebox.com/api-reference/python/tilebox.workflows/WorkflowClient.update



```python theme={"system"}
def WorkflowClient.update(
    workflow_or_slug: Workflow | str,
    name: str | None = None,
    description: str | None = None,
) -> Workflow
```

Update a workflow.

## Parameters

<ParamField type="Workflow | str">
  The workflow or workflow slug to update.
</ParamField>

<ParamField type="str | None">
  Optional new display name for the workflow. If omitted, the display name is unchanged.
</ParamField>

<ParamField type="str | None">
  Optional new workflow description. Pass an empty string to clear the description. If omitted, the description is unchanged.
</ParamField>

## Returns

The updated workflow object.

<RequestExample>
  ```python Python theme={"system"}
  workflow = workflow_client.update(
      "scene-processing",
      name="Scene processing",
      description="Process new scenes into analysis-ready outputs.",
  )
  ```
</RequestExample>


# Authentication
Source: https://docs.tilebox.com/authentication

To access the Tilebox API, you must authenticate your requests. This guide explains how authentication works, focusing on API keys used as bearer tokens.

## Creating an API key

To create an API key, log into the [Tilebox Console](https://console.tilebox.com). Navigate to [Settings -> API Keys](https://console.tilebox.com/settings/api-keys) and click the "Create API Key" button.

<Info>
  Keep your API keys secure. Deactivate any keys if you suspect they have been compromised.
</Info>

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>

## Set up environment variable

Set `TILEBOX_API_KEY` in your shell environment (optional but recommended). That way, the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) and SDK clients automatically pick it up.

The following commands persist the variable for future terminal sessions and also make it available in the current session.

<CodeGroup>
  ```bash Bash theme={"system"}
  echo 'export TILEBOX_API_KEY="<tbx-api-...>"' >> ~/.bashrc
  source ~/.bashrc
  ```

  ```zsh Zsh theme={"system"}
  echo 'export TILEBOX_API_KEY="<tbx-api-...>"' >> ~/.zshrc
  source ~/.zshrc
  ```

  ```fish Fish theme={"system"}
  set -Ux TILEBOX_API_KEY "<tbx-api-...>"
  ```

  ```powershell PowerShell theme={"system"}
  [Environment]::SetEnvironmentVariable("TILEBOX_API_KEY", "<tbx-api-...>", "User")
  $env:TILEBOX_API_KEY = "<tbx-api-...>"
  ```
</CodeGroup>

Open a new terminal after setting the variable to confirm your shell loads it automatically.

## Manual SDK authentication

In case you didn't configure your shell environment, you can also pass your API key as the `token` parameter when creating an instance of the SDK clients.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client as DatasetsClient
  from tilebox.workflows import Client as WorkflowsClient

  datasets_client = DatasetsClient(token="<tbx-api-...>")
  workflows_client = WorkflowsClient(token="<tbx-api-...>")
  ```

  ```go Go theme={"system"}
  import (
  	"github.com/tilebox/tilebox-go/datasets/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  datasetsClient := datasets.NewClient(datasets.WithAPIKey("<tbx-api-...>"))
  workflowsClient := workflows.NewClient(workflows.WithAPIKey("<tbx-api-...>"))
  ```
</CodeGroup>

<Tip>
  If you set your API key as an environment variable named `TILEBOX_API_KEY`, you can skip the token parameter when creating a client.
</Tip>


# Product Updates
Source: https://docs.tilebox.com/changelog

New Tilebox features, product improvements, and workflow capabilities as they ship.

<Update label="June 2026">
  <Frame>
    <img alt="Closed workflow iteration loop from deploy to run to observe to fix" />

    <img alt="Closed workflow iteration loop from deploy to run to observe to fix" />
  </Frame>

  ## Agentic Workflows

  Tilebox Workflows now supports workflow release publishing and cluster deployments. This adds a new release-based execution path to the workflow orchestrator: package a Python workflow project, publish an immutable release, deploy it to a cluster, run it on release runners, and inspect the resulting job logs and execution traces.

  The release path is designed for closed-loop workflow iteration by developers and agents. An agent can edit workflow code, publish a release, deploy it to a development cluster, submit a test job, inspect the failed task logs or spans, apply a fix, and retry or submit the next job without leaving the same command-line workflow.

  A typical iteration looks like this:

  ```bash theme={"system"}
  tilebox workflow publish-release --json
  tilebox workflow deploy-release --latest --target production --json
  tilebox job submit --task tilebox.com/example/ProcessScene --cluster workflow-dev --json
  tilebox job wait <job-id>
  tilebox job logs <job-id> --json
  ```

  ### What changed

  * **Release runners.** Release runners run in an environment you control, watch a cluster, load the deployed releases for that cluster, and execute compatible tasks without rebuilding the runner process for every workflow code change.
  * **Workflow release publishing.** A workflow is now a long-lived object with a stable slug, and each release captures a concrete version of the workflow project, runtime entrypoint, selected files, and discovered task identifiers.
  * **Project-local workflow configuration.** `tilebox.workflow.toml` binds a repository directory to a workflow slug, build inputs, runtime entrypoint, and optional deployment targets, so commands can use the local project as context.
  * **Cluster deployments.** Publishing and deploying are separate steps. A release can be deployed to one or more clusters, and different clusters can run different releases of the same workflow.

  ### Start here

  <Columns>
    <Card title="Iterate on workflow releases with agents" icon="robot" href="/guides/workflows/agentic-workflow-iteration">
      Use an AI coding agent to edit, publish, deploy, run, inspect, and retry workflow releases.
    </Card>

    <Card title="Workflow configuration" icon="file-code" href="/workflows/build-and-deploy/workflow-configuration">
      Configure `tilebox.workflow.toml`, release contents, runtime entrypoints, and deployment targets.
    </Card>
  </Columns>
</Update>

<Update label="May 2026">
  <Frame>
    <img alt="Job Execution Trace View" />

    <img alt="Job Execution Trace View" />
  </Frame>

  ## Workflow Observability

  Tilebox Workflows now includes built-in observability for jobs and runners. Tilebox captures workflow logs, traces, task status, and runner context, then correlates them with jobs and tasks.

  The Console includes a built-in explorer for workflow observability, so you can inspect task logs, trace timing, failures, and runner behavior from the job view.

  See the [Workflow observability documentation](/workflows/run-and-inspect/introduction) for examples and integration options.
</Update>

<Update label="February 2026">
  <Frame>
    <img alt="Optional Tasks" />
  </Frame>

  ## Optional Tasks

  Subtasks can now be marked as **optional** when submitting them. If an optional task fails, the job continues instead of being canceled. Failed optional tasks are marked with the state `FAILED_OPTIONAL`, and any remaining queued sibling tasks in the same optional subtask tree are automatically `SKIPPED`. Tasks that depend on an optional task still execute even if it failed.

  This is useful for workflows where certain steps are not critical and their failure should not prevent the rest of the job from completing.

  See the [optional tasks documentation](/workflows/concepts/tasks#optional-tasks) for details and examples.
</Update>

<Update label="January 2026">
  <Frame>
    <img alt="MCP Server" />
  </Frame>

  ## MCP Server for Datasets and Workflows

  Introducing the [Tilebox MCP server](/agents-and-ai-tools/tilebox-mcp), which provides tools for AI agents to access and interact with Tilebox datasets and workflows.
</Update>

<Update label="November 2025">
  ## Job List View: Complete Redesign and Filtering Improvements

  <Frame>
    <img alt="Job List View Improvements" />
  </Frame>

  The job list view in the Tilebox Console has been completely redesigned:

  * Infinite scrolling for long job lists
  * Improved filtering and search
    * Filter by state
    * Filter by automation
    * Filter by time range
    * Filter by name
    * Or an arbitrary combination of the above
  * Better readability and organization
  * Added progress indicators
  * Added execution stats
  * The same new filter options are also available in our [Language SDKs](/sdks)
</Update>

<Update label="October 2025">
  ## Spatio-Temporal Explorer Redesign

  <Frame>
    <img alt="Console Improvements" />
  </Frame>

  The dataset explorer in the Tilebox Console has received a major upgrade for spatio-temporal datasets:

  * Overhaul of the Explorer view for Spatio-temporal datasets
  * Display datapoint footprints directly on the map
  * Display thumbnails and quicklooks of datapoints directly in the Console
  * Added code snippet for storage access to the `Export as Code` dialog
</Update>

<Update label="September 2025">
  ## Progress Indicators

  <Frame>
    <img alt="Progress Indicators" />
  </Frame>

  User defined progress indicators for jobs are now available. See the [progress documentation](/workflows/run-and-inspect/progress) for more details.
</Update>

<Update label="May 2025">
  ## Go Client

  <Frame>
    <img alt="Tilebox Go Client" />
  </Frame>

  Excited to announce the release of the Tilebox Go client!

  **Features**

  * Datasets client
  * Statically typed dataset types
  * CLI to generate dataset types
  * Workflows client
  * Go runners

  To get started, check out the [Go SDK documentation](/sdks/go/install).
</Update>

<Update label="April 2025">
  ## Spatio-Temporal datasets

  <Frame>
    <img alt="Spatio-Temporal Datasets" />
  </Frame>

  Spatio-temporal datasets are officially out, fully supported in all languages and available as a category to create in custom datasets!

  The core problems that spatio-temporal datasets solve are

  * finding relevant data quickly (e.g. all Sentinel 2 granules along the US coastline, last year),
  * storing auxiliary geographically coded data (e.g. weather station data, ground truth data),
  * cataloging higher level data and results

  [Here's a short video](https://share.descript.com/view/khO3QJslhgU) on performance and core capabilities.

  We're excited about this as cataloging has until now been an unsexy but hard problem, and it's great to finally have a solution out there

  **More information**

  * [Spatio Temporal datasets documentation](https://docs.tilebox.com/datasets/types/spatiotemporal)
  * All open data now supports spatio-temporal queries
  * [Create your own spatio-temporal datasets](/guides/datasets/build-spatiotemporal-catalog)
  * [Ingesting spatio-temporal data](/datasets/ingest)
</Update>

<Update label="March 2025">
  ## Custom Datasets

  <Frame>
    <img alt="Custom datasets" />
  </Frame>

  Create your own custom datasets!

  * statically typed
  * with clients in Python and Go

  Use it to organize anything from telemetry, raw payload metadata, auxiliary sensor data, configuration data, or
  internal data catalogs.

  **Quickstart**

  1. Specify the data type in the Console
  2. Create a collection
  3. Use client.ingest() to ingest a `xarray.Dataset` or `pandas.DataFrame`
  4. Query away!

  For detailed instructions, check out the [Build a spatio-temporal catalog](/guides/datasets/build-spatiotemporal-catalog) how-to guide.
</Update>


# Console
Source: https://docs.tilebox.com/console

A web interface for exploring your datasets and workflows, managing API keys and team members, and keeping track of your usage of the Tilebox platform.

The [Tilebox Console](https://console.tilebox.com) is a web interface that enables you to explore your datasets and workflows, manage your account and API keys, add collaborators to your team and monitor your usage.

## Datasets

The datasets explorer lets you explore available datasets for your team. You can select a dataset, view its collections, and load data for a collection within a specified time range.

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>

When you click a specific event time in the data point list view, a detailed view of that data point will appear.

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>

### Export as Code

After selecting a dataset, collection, and time range, you can export the current selection as a Python code snippet. This will copy a code snippet like the one below to your clipboard.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  datasets = client.datasets()
  sentinel2_msi = datasets.open_data.copernicus.sentinel2_msi
  data = sentinel2_msi.collection("S2A_S2MSI1C").query(
      temporal_extent=("2024-07-12", "2024-07-26"),
      show_progress=True,
  )
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := datasets.NewClient()

  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")
  if err != nil {
      log.Fatalf("Failed to get dataset: %v", err)
  }

  collection, err := client.Collections.Get(ctx, dataset.ID, "S2A_S2MSI1C")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  startDate := time.Date(2024, 7, 12, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2024, 7, 26, 0, 0, 0, 0, time.UTC) 
  timeInterval := query.NewTimeInterval(startDate, endDate)

  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithCollectionIDs(collection.ID),
      datasets.WithTemporalExtent(timeInterval),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

<Tip>
  Paste the snippet into a [notebook](/sdks/python/sample-notebooks) to interactively explore the
  [`xarray.Dataset`](/sdks/python/xarray) that is returned.
</Tip>

## Workflows

The workflows section of the console allows you to view jobs, create clusters and create near real-time automations.

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>

## Account

### API Keys

The API Keys page enables you to manage your API keys. You can create new API keys, revoke existing ones, and view currently active API keys.

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>

### Usage

The Usage page allows you to view your current usage of the Tilebox API.

<Frame>
  <img alt="Tilebox Console" />

  <img alt="Tilebox Console" />
</Frame>


# Collections
Source: https://docs.tilebox.com/datasets/concepts/collections

Collections organize data points within a dataset into logical groups that are commonly queried together, such as groupings by satellite or instrument.

Collections group data points within a dataset. They help represent logical groupings of data points that are commonly queried together.
For example, if your dataset includes data from a specific instrument on different satellites, you can group the data points from each satellite into a collection.

Refer to the examples below for common use cases when working with collections.
These examples assume that you have already created a client and selected a dataset as shown below.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  datasets = client.datasets()
  dataset = datasets.open_data.copernicus.landsat8_oli_tirs
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log"

  	"github.com/tilebox/tilebox-go/datasets/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := datasets.NewClient()

  	dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.landsat8_oli_tirs")
  	if err != nil {
  		log.Fatalf("Failed to get dataset: %v", err)
  	}
  }
  ```
</CodeGroup>

## Listing collections

To list the collections for a dataset, use the `collections` method on the dataset object.

<CodeGroup>
  ```python Python theme={"system"}
  collections = dataset.collections()
  print(collections)
  ```

  ```go Go theme={"system"}
  collections, err := client.Collections.List(ctx, dataset.ID)
  if err != nil {
    log.Fatalf("Failed to list collections: %v", err)
  }

  log.Println(collections)
  ```
</CodeGroup>

```plaintext Output theme={"system"}
{'L1GT': Collection L1GT: [2013-03-25T12:08:43.699 UTC, 2024-08-19T12:57:32.456 UTC] (154288 data points),
 'L1T': Collection L1T: [2013-03-26T09:33:19.763 UTC, 2020-08-24T03:21:50.000 UTC] (87958 data points),
 'L1TP': Collection L1TP: [2013-03-24T00:25:55.457 UTC, 2024-08-19T12:58:20.229 UTC] (322041 data points),
 'L2SP': Collection L2SP: [2015-01-01T07:53:35.391 UTC, 2024-08-12T12:52:03.243 UTC] (191110 data points)}
```

[dataset.collections](/api-reference/python/tilebox.datasets/Dataset.collections) returns a dictionary mapping collection names to their corresponding collection objects. Each collection has a unique name within its dataset.

## Creating collections

To create a collection in a dataset, use [dataset.create\_collection()](/api-reference/python/tilebox.datasets/Dataset.create_collection). This method returns the created collection object.

<CodeGroup>
  ```python Python theme={"system"}
  collection = dataset.create_collection("My-collection")
  ```

  ```go Go theme={"system"}
  collection, err := client.Collections.Create(ctx, dataset.ID, "My-collection")
  ```
</CodeGroup>

You can use [dataset.get\_or\_create\_collection()](/api-reference/python/tilebox.datasets/Dataset.get_or_create_collection) to get a collection by its name. If the collection does not exist, it will be created.

<CodeGroup>
  ```python Python theme={"system"}
  collection = dataset.get_or_create_collection("My-collection")
  ```

  ```go Go theme={"system"}
  collection, err := client.Collections.GetOrCreate(ctx, dataset.ID, "My-collection")
  ```
</CodeGroup>

## Accessing individual collections

Once you have listed the collections for a dataset using [dataset.collections()](/api-reference/python/tilebox.datasets/Dataset.collections), you can access a specific collection by retrieving it from the resulting dictionary with its name. Use [collection.info()](/api-reference/python/tilebox.datasets/Collection.info) in Python or `String()` in Go to get details (name, availability, and count) about it.

<CodeGroup>
  ```python Python theme={"system"}
  collections = dataset.collections()
  terrain_correction = collections["L1GT"]
  collection_info = terrain_correction.info()
  print(collection_info)
  ```

  ```go Go theme={"system"}
  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.landsat8_oli_tirs")
  if err != nil {
    log.Fatalf("Failed to get dataset: %v", err)
  }

  collection, err := client.Collections.Get(ctx, dataset.ID, "L1GT")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  log.Println(collection.String())
  ```
</CodeGroup>

```plaintext Output theme={"system"}
L1GT: [2013-03-25T12:08:43.699 UTC, 2024-08-19T12:57:32.456 UTC] (154288 data points)
```

You can also access a specific collection directly using the [dataset.collection](/api-reference/python/tilebox.datasets/Dataset.collection) method on the dataset object. This method allows you to get the collection without having to list all collections first.

<CodeGroup>
  ```python Python theme={"system"}
  terrain_correction = dataset.collection("L1GT")
  collection_info = terrain_correction.info()
  print(collection_info)
  ```

  ```go Go theme={"system"}
  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.landsat8_oli_tirs")
  if err != nil {
    log.Fatalf("Failed to get dataset: %v", err)
  }

  collection, err := client.Collections.Get(ctx, dataset.ID, "L1GT")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  log.Println(collection.String())
  ```
</CodeGroup>

```plaintext Output theme={"system"}
L1GT: [2013-03-25T12:08:43.699 UTC, 2024-08-19T12:57:32.456 UTC] (154288 data points)
```

## Deleting collections

Collections can be deleted from a dataset using the `delete_collection` method.

To delete a collection, you need to have write permission on the dataset.

<Warning>
  Deleting a collection will delete all data points in the collection.
</Warning>

<CodeGroup>
  ```python Python theme={"system"}
  dataset.delete_collection(collection)
  ```

  ```go Go theme={"system"}
  err := client.Collections.Delete(ctx, dataset.ID, collection.ID)
  ```
</CodeGroup>

## Errors you may encounter

### NotFoundError

If you attempt to access a collection with a non-existent name, a `NotFoundError` is raised. For example:

<CodeGroup>
  ```python Python theme={"system"}
  dataset.collection("Sat-X").info() # raises NotFoundError: 'No such collection Sat-X'
  ```

  ```go Go theme={"system"}
  collection, err := client.Collections.Get(ctx, dataset.ID, "Sat-X")
  if err != nil {
    log.Fatal(err) // prints 'failed to get collections: not_found: no such collection'
  }
  ```
</CodeGroup>

## Next steps

<Columns>
  <Card title="Query Data" icon="server" href="/datasets/query/querying-data">
    Learn how to query data from a collection.
  </Card>
</Columns>


# Datasets
Source: https://docs.tilebox.com/datasets/concepts/datasets

Datasets are strongly typed containers in which every data point within a collection shares the same structure, ensuring consistent and reliable data access.

<Tip>
  You can create your own, Custom Datasets via the [Tilebox Console](/console).
</Tip>

## Related Guides

<Columns>
  <Card title="Build a spatio-temporal catalog" icon="database" href="/guides/datasets/build-spatiotemporal-catalog">
    Learn how to create a custom dataset catalog with the Python SDK.
  </Card>

  <Card title="Ingest into a spatio-temporal catalog" icon="up-from-bracket" href="/guides/datasets/ingest-into-spatiotemporal-catalog">
    Learn how to ingest GeoParquet metadata into an existing spatio-temporal catalog.
  </Card>
</Columns>

## Dataset types

Each dataset is of a specific type. Each dataset type comes with a set of required fields for each data point.
The dataset type also determines the query capabilities for a dataset, for example, whether a dataset supports time-based queries
or additionally also spatially filtered queries.

To find out which fields are required for each dataset type check out the documentation for the available dataset types
below.

<Columns>
  <Card title="Timeseries Data" icon="diamonds-4" href="/datasets/types/timeseries">
    Each data point is linked to a specific point in time. Common for satellite telemetry, or other time-based data.
    Supports efficient time-based queries.
  </Card>

  <Card title="Spatio-temporal Data" icon="earth-europe" href="/datasets/types/spatiotemporal">
    Each data point is linked to a specific point in time and a location on the Earth's surface. Common for satellite
    imagery. Supports efficient time-based and spatially filtered queries.
  </Card>
</Columns>

## Dataset specific fields

Additionally, each dataset has a set of fields that are specific to that dataset. Fields are defined during dataset
creation. That way, all data points in a dataset are strongly typed and are validated during ingestion.
The required fields of the dataset type, as well as the custom fields specific to each dataset together make up the
**dataset schema**.

Once a **dataset schema** is defined, existing fields cannot be removed or edited as soon as data has been ingested into it.
You can always add new fields to a dataset, since all fields are always optional.

<Tip>
  The only exception to this rule are empty datasets. If you empty all collections in a dataset, you can freely
  edit the data schema, since no conflicts with existing data points can occur.
</Tip>

## Field types

When defining the data schema, you can specify each field's type. The following field types are supported.

### Primitives

| Type                 | Description                                 | Example value |
| -------------------- | ------------------------------------------- | ------------- |
| <span>string</span>  | A string of characters of arbitrary length. | `Some string` |
| <span>int64</span>   | A 64-bit signed integer.                    | `123`         |
| <span>uint64</span>  | A 64-bit unsigned integer.                  | `123`         |
| <span>float64</span> | A 64-bit floating-point number.             | `123.45`      |
| <span>bool</span>    | A boolean.                                  | `true`        |
| <span>bytes</span>   | A sequence of arbitrary length bytes.       | `0xAF1E28D4`  |

### Time

| Type                   | Description                                                                                                                                                                                                                      | Example value          |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| <span>Duration</span>  | A signed, fixed-length span of time represented as a count of seconds and fractions of seconds at nanosecond resolution. See [Duration](https://protobuf.dev/reference/protobuf/google.protobuf/#duration) for more information. | `12s 345ms`            |
| <span>Timestamp</span> | A point in time, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time. See [Timestamp](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp) for more information.           | `2023-05-17T14:30:00Z` |

### Identifier

| Type              | Description                                                                                            | Example value                          |
| ----------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------- |
| <span>UUID</span> | A [universally unique identifier (UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier). | `126a2531-c98d-4e06-815a-34bc5b1228cc` |

### Geospatial

| Type                  | Description                                                               | Example value                           |
| --------------------- | ------------------------------------------------------------------------- | --------------------------------------- |
| <span>Geometry</span> | Geospatial geometries of type Point, LineString, Polygon or MultiPolygon. | `POLYGON ((12.3 -5.4, 12.5 -5.4, ...))` |

### Arrays

Every type is also available as an array, allowing to ingest multiple values of the underlying type for each data point. The size of the array is flexible, and can be different for each data point.

## Listing datasets

You can use [your client instance](/datasets/introduction#creating-a-datasets-client) to access the datasets available to you. To list all available datasets, use the `datasets` method of the client.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  datasets = client.datasets()
  print(datasets)
  ```

  ```go Go theme={"system"}
  package main

  import (
    "context"
    "log"

    "github.com/tilebox/tilebox-go/datasets/v1"
  )

  func main() {
    ctx := context.Background()

    client := datasets.NewClient()

    allDatasets, err := client.Datasets.List(ctx)
    if err != nil {
      log.Fatalf("Failed to list datasets: %v", err)
    }

    for _, dataset := range allDatasets {
      log.Println(dataset)
    }
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
open_data:
    asf:
        ers_sar: European Remote Sensing Satellite (ERS) Synthetic Aperture Radar (SAR) Granules
    copernicus:
        landsat8_oli_tirs: Landsat-8 is part of the long-running Landsat programme ...
        sentinel1_sar: The Sentinel-1 mission is the European Radar Observatory for the ...
        sentinel2_msi: Sentinel-2 is equipped with an optical instrument payload that samples ...
        sentinel3_olci: OLCI (Ocean and Land Colour Instrument) is an optical instrument used to ...
        ...
```

Once you have your dataset object, you can use it to [list the available collections](/datasets/concepts/collections) for the dataset.

<Tip>
  In python, if you're using an IDE or an interactive environment with auto-complete, you can use it on your client instance to discover the datasets available to you. Type `client.` and trigger auto-complete after the dot to do so.
</Tip>

## Creating / Updating a dataset

You can create a dataset using one of the available [client SDKs](/sdks/introduction), or use the [Tilebox Console](/console) when you prefer a visual schema editor.

<CodeGroup>
  ```python Python theme={"system"}
  from datetime import timedelta
  from tilebox.datasets import Client
  from tilebox.datasets.data.datasets import DatasetKind

  client = Client()

  dataset = client.create_or_update_dataset(
      DatasetKind.SPATIOTEMPORAL,
      "my_landsat8_oli_tirs",
      [
          {
              "name": "granule_name",
              "type": str,
              "description": "The name of the Landsat-8 granule.",
              "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1",
          },
          {
              "name": "cloud_cover",
              "type": float,
              "description": "Cloud cover percentage",
              "example_value": "91",
          },
          {
              "name": "proj_shape",
              "type": list[int],
              "description": "Raster shape",
              "example_value": "[6971, 7801]",
          },
      ],
      name="Personal Landsat-8 catalog",
  )
  ```

  ```go Go theme={"system"}
  dataset, err := client.Datasets.Create(ctx,
    datasets.KindSpatiotemporal,
    "my_landsat8_oli_tirs",
    "Personal Landsat-8 catalog",
    []datasets.Field{
      field.String("granule_name").Description("The name of the Earth View Granule").ExampleValue("20220830_185202_SN18_10N_552149_5297327"),
      field.Float64("cloud_cover").Description("Cloud cover percentage").ExampleValue("91"),
      field.Int64("proj_shape").Repeated().Description("Raster shape").ExampleValue("[6971, 7801]"),
    }
  )
  ```
</CodeGroup>

This code will create a new catalog with 7 fields in total.
4 of those fields are auto-generated by choosing the spatio-temporal dataset type, and 3 (`granule_name`, `cloud_cover`, `proj_shape`) are custom fields that are defined.

<Tip>
  If a dataset with the same `code_name` already exists, it will be updated instead.
</Tip>

## Accessing a dataset

Each dataset has an automatically generated *slug* that can be used to access it. The *slug* is the name of the group, followed by a dot, followed by the dataset *code name*.
For example, the *slug* for the Sentinel-2 MSI dataset, which is part of the `open_data.copernicus` group, is `open_data.copernicus.sentinel2_msi`.

To access a dataset, use the `dataset` method of your client instance and pass the *slug* of the dataset as an argument.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  s2_msi_dataset = client.dataset("open_data.copernicus.sentinel2_msi")
  ```

  ```go Go theme={"system"}
  s2MsiDataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")
  ```
</CodeGroup>

Once you have your dataset object, you can use it to [access available collections](/datasets/concepts/collections) for the dataset.

## Deleting a dataset

Datasets can be deleted through the [Tilebox Console](/console) by clicking the `Delete` button in the dataset page.

Empty datasets will be deleted right away. A dataset is considered empty if none of its collections contain any data points.

A non-empty dataset can also be deleted, but is a destructive operation.
Every data point in every collection of the dataset will be deleted.

As a safety measure, Tilebox soft-deletes the dataset for 7 days before permanently deleting it.
Please [get in touch](mailto:support@tilebox.com) if you deleted a dataset by accident and want to restore it.


# Deleting Data
Source: https://docs.tilebox.com/datasets/delete

Remove individual datapoints from a dataset collection by specifying their unique identifiers, or delete many at once by selecting an entire time range.

<Note>
  You need to have write permission on the collection to be able to delete datapoints.
</Note>

Check out the examples below for common scenarios of deleting data from a collection.

## Deleting data by datapoint IDs

To delete data from a collection, use the [delete](/api-reference/python/tilebox.datasets/Collection.delete) method. This method accepts a list of datapoint IDs to delete.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  datasets = client.datasets()
  collections = datasets.my_custom_dataset.collections()
  collection = collections["Sensor-1"]

  n_deleted = collection.delete([
    "0195c87a-49f6-5ffa-e3cb-92215d057ea6",
    "0195c87b-bd0e-3998-05cf-af6538f34957",
  ])

  print(f"Deleted {n_deleted} data points.")
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log"
  	"log/slog"

  	"github.com/google/uuid"
  	"github.com/tilebox/tilebox-go/datasets/v1"
  )

  func main() {
  	ctx := context.Background()

  	client := datasets.NewClient()

  	dataset, err := client.Datasets.Get(ctx, "my_custom_dataset")
  	if err != nil {
  		log.Fatalf("Failed to get dataset: %v", err)
  	}

  	collection, err := client.Collections.Get(ctx, dataset.ID, "Sensor-1")
  	if err != nil {
  		log.Fatalf("Failed to create collection: %v", err)
  	}

  	datapointIDs := []uuid.UUID{
  		uuid.MustParse("0195c87a-49f6-5ffa-e3cb-92215d057ea6"),
  		uuid.MustParse("0195c87b-bd0e-3998-05cf-af6538f34957"),
  	}
  	numDeleted, err := client.Datapoints.DeleteIDs(ctx, collection.ID, datapointIDs)
  	if err != nil {
  		log.Fatalf("Failed to delete datapoints: %v", err)
  	}
  	slog.Info("Deleted data points", slog.Int64("deleted", numDeleted))
  }
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  Deleted 2 data points.
  ```
</CodeGroup>

<Tip>
  In python, `delete` not only takes a list of datapoint IDs as string, but supports a wide range of other useful input types as well.
  See the [delete](/api-reference/python/tilebox.datasets/Collection.delete) API documentation for more details.
</Tip>

### Possible errors

* `NotFoundError`: raised if one of the data points is not found in the collection. If any of the data points are not found,
  nothing will be deleted
* `ValueError`: raised if one of the specified ids is not a valid UUID

## Deleting a time interval

One common way to delete all datapoints in a time interval is to first query it from a collection and then deleting those
found datapoints. For this use case it often is a good idea to query the datapoints with `skip_data=True` to avoid actually
loading the data fields, since only the datapoint IDs are required. See [skipping data fields](/datasets/query#skipping-data-fields) for more details.

<CodeGroup>
  ```python Python theme={"system"}
  to_delete = collection.query(temporal_extent=("2023-05-01", "2023-06-01"), skip_data=True)

  n_deleted = collection.delete(to_delete)
  print(f"Deleted {n_deleted} data points.")
  ```

  ```go Go theme={"system"}
  datasetID := uuid.MustParse("25a6f262-f6eb-4de5-be4f-b021f4f7dd13")
  collectionID := uuid.MustParse("c5145c99-1843-4816-9221-970f9ce3ac93")
  startDate := time.Date(2023, time.May, 1, 0, 0, 0, 0, time.UTC)
  endDate := time.Date(2023, time.June, 1, 0, 0, 0, 0, time.UTC)
  mai2023 := query.NewTimeInterval(startDate, endDate)

  var toDelete []*v1.Sentinel2Msi
  err := client.Datapoints.QueryInto(ctx,
    datasetID,
    &toDelete,
    datasets.WithCollectionIDs(collectionID),
    datasets.WithTemporalExtent(mai2023),
    datasets.WithSkipData(),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  numDeleted, err := client.Datapoints.Delete(ctx, collectionID, toDelete)
  if err != nil {
    log.Fatalf("Failed to delete datapoints: %v", err)
  }
  slog.Info("Deleted data points", slog.Int64("deleted", numDeleted))
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  Deleted 104 data points.
  ```
</CodeGroup>

## Automatic batching

Tilebox automatically batches the delete requests for you, so you don't have to worry about the maximum request size.


# Working with Geometries
Source: https://docs.tilebox.com/datasets/geometries

Best practices for handling geometries in Tilebox, covering winding order conventions and reference systems as well as edge cases like antimeridian crossings.

Geometries are a common cause of friction in geospatial data processing, especially when working with shapes that cross the antimeridian or cover a pole.
Tilebox minimizes this friction, but adhering to the following best practices ensures optimal results. In doing so, you can ensure that no geometry related issues will
arise even when interfacing with other libraries and tools that may not properly support non-linearities in geometries.

## Best Practices

To ensure expected behavior when working with geometries, it's recommended to follow these best practices:

1. Use [language specific geometry libraries](#geometry-libraries-and-common-formats) for creating, parsing and serializing geometries.
2. Define polygon exterior rings with a counter-clockwise [winding order](#winding-order).
3. Define polygon holes (interior rings) with a clockwise [winding order](#winding-order).
4. Ensure longitude values are within `[-180, 180]` and latitude values are within `[-90, 90]`.
5. For geometries crossing the antimeridian, [cut them along the antimeridian](#antimeridian-cutting) into two parts: one for the eastern hemisphere and one for the western.
6. Define geometries covering a pole in [a specific way](#pole-coverings) for correct handling.
7. When downloading geometries from external sources (for example from STAC Catalogs), always verify these assumptions to prevent unexpected behavior before ingesting them into Tilebox.

## Geometry vs Geography

Some systems distinguish between `Geometry` and `Geography` data types. The primary distinction lies in how edge interpolation is handled along the antimeridian. `Geometry` represents a shape in an arbitrary 2D Cartesian coordinate system and does not perform edge interpolation.
In contrast, `Geography` typically wraps around the antimeridian by performing edge interpolation, such as converting Cartesian coordinates to a 3D spherical coordinate system. Typically, Geography types limit `x` coordinates to `[-180, 180]` and `y` coordinates to `[-90, 90]`, whereas `Geometry` types have no such limitations.

Tilebox does not differentiate between `Geometry` and `Geography` types. Instead, it offers a [query option to specify a coordinate reference system](/datasets/query/filter-by-location#coordinate-reference-system). This controls whether geometry intersections and containment checks are performed in a 3D spherical coordinate system (which correctly handles antimeridian crossings) or a standard 2D Cartesian lat/lon coordinate system.

## Terminology

Familiarize yourself with key geometry concepts.

<Tip>
  A great resource to learn more about these concepts is this [blog post by Tom MacWright](https://macwright.com/2015/03/23/geojson-second-bite).
</Tip>

**Point**

A `Point` is a specific location on the Earth's surface, represented by a `longitude` and `latitude` coordinate.

**Ring**

A `Ring` is a collection of points that form a closed loop. The order of the points within the ring matter, since that defines the [winding order](#winding-order) of the ring, which is either clockwise or counter-clockwise.
Every ring should be explicitly closed, meaning that the first and last point should be the same.

**Polygon**

A `Polygon` is a collection of rings. The first ring, the `exterior ring` defines the polygon's boundary. Any other rings are called `interior rings`, representing holes within the polygon.

**MultiPolygon**

A `MultiPolygon` is a collection of polygons.

## Geometry libraries and common formats

Geometries can be expressed in different formats. The [Tilebox Client SDKs](/sdks/introduction) delegate geometry handling to external, well-known libraries.

| Client SDK | Geometry library used by Tilebox                     |
| ---------- | ---------------------------------------------------- |
| Python     | [Shapely](https://shapely.readthedocs.io/en/stable/) |
| Go         | [Orb](https://github.com/paulmach/orb)               |

Here is an example of how to define a `Polygon` geometry, covering the area of the state of Colorado using Tilebox Client SDKs.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import Polygon

  area = Polygon(
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )
  ```

  ```go Go theme={"system"}
  // import "github.com/paulmach/orb"

  area := orb.Polygon{
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }
  ```
</CodeGroup>

These libraries also enable Tilebox to support common geometry formats.

### GeoJSON

[GeoJSON](https://datatracker.ietf.org/doc/html/rfc7946) is a geospatial data interchange format based on JavaScript Object Notation (JSON).

```json colorado.geojson theme={"system"}
{
    "type": "Polygon",
    "coordinates": [
        [
            [-109.05, 41.0],
            [-109.045, 37.0],
            [-102.05, 37.0],
            [-102.05, 41.0],
            [-109.05, 41.0]
        ]
    ]
}
```

Reading such a GeoJSON geometry can be achieved as follows.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import from_geojson

  with open("colorado.geojson", "r") as f:
      area = from_geojson(f.read())
  ```

  ```go Go theme={"system"}
  import (
      "github.com/paulmach/orb/geojson"
      "os"
  )

  func readGeoJSON() (orb.Geometry, error) {
      geometryData, err := os.ReadFile("colorado.geojson")
  	if err != nil {
  		return nil, err
  	}
  	geojsonGeometry, err := geojson.UnmarshalGeometry(geometryData)
  	if err != nil {
  		return nil, err
  	}
  	return geojsonGeometry.Geometry(), nil
  }
  ```
</CodeGroup>

### Well-Known Text (WKT)

[Well-Known Text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) is a text markup language for representing vector geometry objects.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import from_wkt

  wkt = "POLYGON ((-109.05 41, -109.045 37, -102.05 37, -102.05 41, -109.05 41))"
  area = from_wkt(wkt)
  ```

  ```go Go theme={"system"}
  import (
      "github.com/paulmach/orb/encoding/wkt"
  )

  func readWKT() (orb.Geometry, error) {
  	wktGeometry := "POLYGON ((-109.05 41, -109.045 37, -102.05 37, -102.05 41, -109.05 41))"

  	return wkt.Unmarshal(wktGeometry)
  }
  ```
</CodeGroup>

### Well-Known Binary (WKB)

[Well-Known Binary (WKB)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary) is a binary representation of vector geometry objects, a binary equivalent to WKT.

Given a `colorado.wkb` file containing a binary encoding of the Colorado polygon, it can be read as follows.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import from_wkb

  with open("colorado.wkb", "rb") as f:
      area = from_wkb(f.read())
  ```

  ```go Go theme={"system"}
  import (
      "github.com/paulmach/orb/encoding/wkb"
      "os"
  )

  func readWKB() (orb.Geometry, error) {
  	binaryData, err := os.ReadFile("colorado.wkb")
  	if err != nil {
  		return nil, err
  	}
  	return wkb.Unmarshal(binaryData)
  }
  ```
</CodeGroup>

<Tip>
  There is also an extended well known binary format (ewkb) that supports additional information such as specifying a spatial reference system (like EPSG:4326) in the encoded geometry.
  Pythons `shapely` library supports this out of the box, and the `orb` library for Go supports it as well via the `github.com/paulmach/orb/encoding/ewkb` package.
</Tip>

## Winding Order

A ring's winding order defines its orientation (clockwise or counter-clockwise), determined by the sequence of its points.

Geometries in Tilebox follow the right hand rule defined by the [GeoJSON specification](https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.6):

> A linear ring MUST follow the right-hand rule with respect to the area it bounds, i.e., exterior rings are counterclockwise, and holes are clockwise.

Thus, polygon exterior rings must have a counter-clockwise winding order, and interior rings must have a clockwise winding order. Failing to follow this rule can lead to unexpected query results, as shown below.

Take the following `Polygon` consisting of the same exterior ring but in different winding orders.

<Columns>
  ```plaintext Counter-clockwise theme={"system"}
  POLYGON (
    (-5 56, -6 29, 14 28, 23 55, -5 56)
  )
  ```

  ```plaintext Clockwise theme={"system"}
  POLYGON (
    (-5 56, 23 55, 14 28, -6 29, -5 56)
  )
  ```
</Columns>

This is how those two geometries would be interpreted on a sphere.

<Columns>
  <Frame>
    <img alt="Polygon over Europe with correct winding order" />
  </Frame>

  <Frame>
    <img alt="Polygon covering the whole globe with a hole over Europe due to incorrect winding order" />
  </Frame>
</Columns>

<Tip>
  Tilebox automatically detects and corrects incorrect winding orders during data ingestion.
</Tip>

Since such unexpected behavior can cause issues and be hard to debug, Tilebox detects incorrect winding orders and automatically fixes them during ingestion.
This means that geometries covering nearly the entire globe except for a small holes **cannot** be ingested into Tilebox by simply specifying them as a `Polygon` with an exterior ring in clockwise winding order.

Instead, such geometries should be defined as a `Polygon` consisting of two rings:

* an exterior ring covering the whole globe in counter-clockwise winding order
* an interior ring specifying the hole in clockwise winding order

Such a definition, as shown below, ensures correct interpretation by Tilebox, and is also valid according to the [GeoJSON specification](https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.6).

```plaintext Polygon covering the whole globe with a hole over Europe theme={"system"}
POLYGON (
  (-180 90, -180 -90, 180 -90, 180 90, -180 90),
  (-5 56, 23 55, 14 28, -6 29, -5 56)
)
```

To verify the winding order of a geometry, you can use the following code snippets.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import Polygon

  polygon = Polygon(
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )
  print(polygon.exterior.is_ccw)  # True
  ```

  ```go Go theme={"system"}
  import (
      "github.com/paulmach/orb"
  )

  func isCounterClockwise() bool {
      poly := orb.Polygon{
          {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
      }
  	exterior := poly[0]
  	if exterior.Orientation() == orb.CCW {
  		return true
  	}
  	return false
  }

  ```
</CodeGroup>

## Antimeridian Crossings

Geometries that cross the antimeridian are a common in satellite data, but can cause issues when not handled correctly.

Take the following `Polygon` that crosses the antimeridian.

<Frame>
  <img alt="Polygon crossing the antimeridian" />

  <img alt="Polygon crossing the antimeridian" />
</Frame>

Below are a couple of different ways to express such a geometry.

```plaintext Using exact longitude / latitude coordinates for each point theme={"system"}
POLYGON ((173 5, -175 3, -172 20, 177 23, 173 5))
```

While valid, this representation can cause issues for many libraries performing calculations or visualizations in a Cartesian coordinate system. The line segment from `lon=173` to `lon=-175` is interpreted as a `348-degree` line crossing the null meridian, spanning nearly the entire globe.
Visualizations of such geometries often appear as shown below.

<Frame>
  <img alt="Polygon crossing the antimeridian" />

  <img alt="Polygon crossing the antimeridian" />
</Frame>

To mitigate these issues, some Cartesian coordinate tools extend the longitude range beyond the usual `[-180, 180]` bounds.
Expressing the geometry in such a way, may look like this.

```plaintext Extending the longitude range beyond 180 theme={"system"}
POLYGON ((173 5, 185 3, 188 20, 177 23, 173 5))
```

Or, the exact same area on the globe can also be expressed as a geometry by extending the longitude range below `-180`.

```plaintext Extending the longitude range below -180 theme={"system"}
POLYGON ((-187 5, -175 3, -172 20, -183 23, -187 5))
```

While most visualization tools may handle such geometries correctly, special care is required for intersection and containment checks.
The following code snippet demonstrates this by constructing a Polygon that covers the same area using both methods and checking for intersection (which should logically occur, as they represent the same geographic area).

```python Incorrect intersection check theme={"system"}
from shapely import from_wkt

a = from_wkt("POLYGON ((173 5, 185 3, 188 20, 177 23, 173 5))")
b = from_wkt("POLYGON ((-187 5, -175 3, -172 20, -183 23, -187 5))")

print(a.intersects(b))  # False
```

### Antimeridian Cutting

Additionally, none of the representations shown are valid according to the GeoJSON specification. The GeoJSON specification does offers a solution for this problem, though, which is [Antimeridian Cutting](https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.9).
It suggests always cutting lines and polygons that cross the antimeridian into two parts—one for the eastern hemisphere and one for the western hemisphere.

To achieve this, check out the [antimeridian](https://pypi.org/project/antimeridian/) python package, and an implementation of it in [Go](https://github.com/go-geospatial/antimeridian).

<Tip>
  The `antimeridian` python package also is a great resource for [learning more about possible antimeridian issues](https://www.gadom.ski/antimeridian/latest/), how the [cutting algorithm](https://www.gadom.ski/antimeridian/latest/the-algorithm/) works and [edge cases](https://www.gadom.ski/antimeridian/latest/failure-modes/) to be aware of.
</Tip>

```plaintext Cutting the polygon along the antimeridian theme={"system"}
MULTIPOLYGON (
    ((180 22.265994, 177 23, 173 5, 180 3.855573, 180 22.265994)),
    ((-180 3.855573, -175 3, -172 20, -180 22.265994, -180 3.855573))
)
```

By cutting geometries along the antimeridian:

* they conform to the GeoJSON specification
* there is no ambiguity about how to correctly interpret the geometry
* they are correctly handled by most visualization tools
* intersection and containment checks yield correct results no matter the [coordinate reference system](/datasets/query/filter-by-location#coordinate-reference-system) used
* all longitude values are within the valid `[-180, 180]` range, making it easier to work with them

## Pole Coverings

Geometries covering a pole can be particularly challenging to handle correctly. No single algorithm addresses all cases, and different libraries and tools may interpret them differently.
Often, providing prior knowledge, such as confirming that a geometry is intended to cover a pole, can resolve these issues.
For an example of this, check out the relevant section in the [antimeridian documentation](https://www.gadom.ski/antimeridian/latest/failure-modes/#force-the-geometry-over-the-north-pole).

Generally speaking though, there are two approaches that work well:

### Approach 1: Cutting out a hole

Define a geometry with an exterior ring that covers the whole globe. Then, cut out a hole by defining an interior ring of the area that not covered by the geometry.

<Tip>
  This approach works especially well for geometries that cover both poles, since then the interior ring is guaranteed to not cover any poles.
</Tip>

<Columns>
  <Frame>
    <img alt="Polygon covering both poles, viewing north pole covering" />

    <img alt="Polygon covering both poles, viewing north pole covering" />
  </Frame>

  <Frame>
    <img alt="Polygon covering both poles, viewing south pole covering" />

    <img alt="Polygon covering both poles, viewing south pole covering" />
  </Frame>
</Columns>

```plaintext Polygon covering both poles theme={"system"}
POLYGON (
    (-180 90, -180 -90, 180 -90, 180 90, -180 90), 
    (-133 72, 153 77, 130 -64, -160 -66, -133 72)
)
```

### Approach 2: Splitting into multiple parts

Another approach, effective for circular caps covering a pole, involves cutting the geometry into multiple parts. Instead of splitting the geometry only at the antimeridian, also split it at the null meridian, and the `90` and `-90` meridians.
For a circular cap covering the north pole, this results in four triangular parts, which can be combined into a `MultiPolygon`. Visualization libraries and intersection/containment checks performed in cartesian coordinate space will typically handle such a geometry correctly.

<Frame>
  <img alt="Polygon covering a pole" />

  <img alt="Polygon covering a pole" />
</Frame>

```plaintext Geometry of a circular cap covering the north pole theme={"system"}
MULTIPOLYGON (
    ((-90 75, 0 75, 0 90, -90 90, -90 75)),
    ((0 75, 90 75, 90 90, 0 90, 0 75)),
    ((90 75, 180 75, 180 90, 90 90, 90 75)),
    ((-180 75, -90 75, -90 90, -180 90, -180 75))
)
```


# Ingesting Data
Source: https://docs.tilebox.com/datasets/ingest

Populate your dataset collections by defining schemas, preparing structured data points, and submitting them efficiently to Tilebox for storage and querying.

<Note>
  You need to have write permission on the collection to be able to ingest data.
</Note>

Check out the examples below for common scenarios of ingesting data into a collection.

## Dataset schema

Tilebox Datasets are strongly typed. This means you can only ingest data that matches the schema of a dataset. The schema is defined during dataset creation time.

The examples on this page assume that you have access to a [Timeseries dataset](/datasets/types/timeseries) that has the following schema:

<Accordion title="MyCustomDataset schema">
  <Tip>
    Check out the [Build a spatio-temporal catalog](/guides/datasets/build-spatiotemporal-catalog) guide for an example of how to create such a dataset.
  </Tip>

  **MyCustomDataset schema**

  | Field name       | Type                         | Description                                                                                         |
  | ---------------- | ---------------------------- | --------------------------------------------------------------------------------------------------- |
  | `time`           | <span>Timestamp</span>       | Timestamp of the data point. Required by the [Timeseries dataset](/datasets/types/timeseries) type. |
  | `id`             | <span>UUID</span>            | Auto-generated UUID for each datapoint.                                                             |
  | `ingestion_time` | <span>Timestamp</span>       | Auto-generated timestamp for when the data point was ingested into the Tilebox API.                 |
  | `value`          | <span>float64</span>         | A numeric measurement value.                                                                        |
  | `sensor`         | <span>string</span>          | A name of the sensor that generated the data point.                                                 |
  | `precise_time`   | <span>Timestamp</span>       | A precise measurement time in nanosecond precision.                                                 |
  | `sensor_history` | <span>Array\[float64]</span> | The last few measurements of the sensor.                                                            |

  <Tip>
    A full overview of available data types can be found in the [here](/datasets/concepts/datasets#field-types).
  </Tip>
</Accordion>

Once you've defined the schema and created a dataset, you can access it and create a collection to ingest data into.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  dataset = client.dataset("my_org.my_custom_dataset")
  collection = dataset.get_or_create_collection("Measurements")
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log"

  	"github.com/tilebox/tilebox-go/datasets/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := datasets.NewClient()

  	dataset, err := client.Datasets.Get(ctx, "my_org.my_custom_dataset")
  	if err != nil {
  		log.Fatalf("Failed to get dataset: %v", err)
  	}

  	collection, err := client.Collections.GetOrCreate(ctx, dataset.ID, "Measurements")
  	if err != nil {
  		log.Fatalf("Failed to get collection: %v", err)
  	}
  }
  ```
</CodeGroup>

## Preparing data for ingestion

Ingestion can be done either in Python or Go.

### Python

[`collection.ingest`](/api-reference/python/tilebox.datasets/Collection.ingest) supports a wide range of input types. Below is an example of using either a `pandas.DataFrame` or an `xarray.Dataset` as input.

#### pandas DataFrame

A [pandas.DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) is a representation of two-dimensional, potentially heterogeneous tabular data. It's a powerful tool for working with structured data, and Tilebox supports it as input for `ingest`.

The example below shows how to construct a `pandas.DataFrame` from scratch, that matches the schema of the `MyCustomDataset` dataset and can be ingested into it.

<CodeGroup>
  ```python Python theme={"system"}
  import pandas as pd

  data = pd.DataFrame({
      "time": [
        "2025-03-28T11:44:23Z",
        "2025-03-28T11:45:19Z",
      ],
      "value": [45.16, 273.15],
      "sensor": ["A", "B"],
      "precise_time": [
        "2025-03-28T11:44:23.345761444Z",
        "2025-03-28T11:45:19.128742312Z",
      ],
      "sensor_history": [
        [-12.15, 13.45, -8.2, 16.5, 45.16],
        [300.16, 280.12, 273.15],
      ],
  })
  print(data)
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
                     time   value sensor                    precise_time                      sensor_history
  0  2025-03-28T11:44:23Z   45.16      A  2025-03-28T11:44:23.345761444Z  [-12.15, 13.45, -8.2, 16.5, 45.16]
  1  2025-03-28T11:45:19Z  273.15      B  2025-03-28T11:45:19.128742312Z            [300.16, 280.12, 273.15]
  ```
</CodeGroup>

Once you have the data ready in this format, you can `ingest` it into a collection.

<CodeGroup>
  ```python Python theme={"system"}
  # now that we have the data frame in the correct format
  # we can ingest it into the Tilebox dataset
  collection.ingest(data)

  # To verify it now contains the 2 data points
  print(collection.info())
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  Measurements: [2025-03-28T11:44:23.000 UTC, 2025-03-28T11:45:19.000 UTC] (2 data points)
  ```
</CodeGroup>

<Tip>
  You can now also head on over to the [Tilebox Console](/console) and view the newly ingested data points there.
</Tip>

#### xarray Dataset

[`xarray.Dataset`](/sdks/python/xarray) is the default format in which Tilebox Datasets returns data when
[querying data](/datasets/query/querying-data) from a collection.
Tilebox also supports it as input for ingestion. The example below shows how to construct an `xarray.Dataset`
from scratch, that matches the schema of the `MyCustomDataset` dataset and can then be ingested into it.
To learn more about `xarray.Dataset`, visit Tilebox dedicated [Xarray documentation page](/sdks/python/xarray).

<CodeGroup>
  ```python Python theme={"system"}
  import pandas as pd

  data = xr.Dataset({
      "time": ("time", [
        "2025-03-28T11:46:13Z",
        "2025-03-28T11:46:54Z",
      ]),
      "value": ("time", [48.1, 290.12]),
      "sensor": ("time", ["A", "B"]),
      "precise_time": ("time", [
        "2025-03-28T11:46:13.345761444Z",
        "2025-03-28T11:46:54.128742312Z",
      ]),
      "sensor_history": (("time", "n_sensor_history"), [
        [13.45, -8.2, 16.5, 45.16, 48.1],
        [280.12, 273.15, 290.12, np.nan, np.nan],
      ]),
  })
  print(data)
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  <xarray.Dataset> Size: 504B
  Dimensions:         (time: 2, n_sensor_history: 5)
  Coordinates:
    * time            (time) <U20 160B '2025-03-28T11:46:13Z' '2025-03-28T11:46...
  Dimensions without coordinates: n_sensor_history
  Data variables:
      value           (time) float64 16B 48.1 290.1
      sensor          (time) <U1 8B 'A' 'B'
      precise_time    (time) <U30 240B '2025-03-28T11:46:13.345761444Z' '2025-0...
      sensor_history  (time, n_sensor_history) float64 80B 13.45 -8.2 ... nan nan
  ```
</CodeGroup>

<Note>
  Array fields manifest in xarray using an extra dimension, in this case `n_sensor_history`. In case
  of different array sizes for each data point, remaining values are filled up with a fill value, depending on the
  `dtype` of the array. For `float64` this is `np.nan` (not a number).
  Don't worry - when ingesting data into a Tilebox dataset, Tilebox will automatically skip those padding fill values
  and not store them in the dataset.
</Note>

Now that you have the `xarray.Dataset` in the correct format, you can ingest it into the Tilebox dataset collection.

<CodeGroup>
  ```python Python theme={"system"}
  collection = dataset.get_or_create_collection("OtherMeasurements")
  collection.ingest(data)

  # To verify it now contains the 2 data points
  print(collection.info())
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  OtherMeasurements: [2025-03-28T11:46:13.000 UTC, 2025-03-28T11:46:54.000 UTC] (2 data points)
  ```
</CodeGroup>

### Go

[`Client.Datapoints.Ingest`](/api-reference/go/datasets/Datapoints.Ingest) supports ingestion of data points in the form of a slice of protobuf messages.

#### Protobuf

Protobuf is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data.

More details on protobuf can be found in the [protobuf section](/sdks/go/protobuf).

In the example below, the `v1.Modis` type has been generated with `tilebox dataset generate`, as described in the [protobuf section](/sdks/go/protobuf).

```go Go theme={"system"}
datapoints := []*v1.Modis{
  v1.Modis_builder{
    Time:        timestamppb.New(time.Now()),
    GranuleName: proto.String("Granule 1"),
  }.Build(),
  v1.Modis_builder{
    Time:        timestamppb.New(time.Now().Add(-5 * time.Hour)),
    GranuleName: proto.String("Past Granule 2"),
  }.Build(),
}

ingestResponse, err := client.Datapoints.Ingest(ctx,
    collectionID,
    &datapoints
    false,
)
```

## Copying or moving data

Since `ingest` takes `query`'s output as input, you can easily copy or move data from one collection to another.

<Tip>
  Copying data like this also works across datasets in case the dataset schemas are compatible.
</Tip>

<CodeGroup>
  ```python Python theme={"system"}
  src_collection = dataset.collection("Measurements")
  data_to_copy = src_collection.query(temporal_extent=("2025-03-28", "2025-03-29"))

  dest_collection = dataset.collection("OtherMeasurements")
  dest_collection.ingest(data_to_copy)  # copy the data to the other collection

  # To verify it now contains 4 datapoints (2 we ingested already, and 2 we copied just now)
  print(dest_collection.info())
  ```

  ```go Go theme={"system"}
  dataset, err := client.Datasets.Get(ctx, "my_org.my_custom_dataset")
  if err != nil {
    log.Fatalf("Failed to get dataset: %v", err)
  }

  srcCollection, err := client.Collections.GetOrCreate(ctx, dataset.ID, "Measurements")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  startDate := time.Date(2025, time.March, 28, 0, 0, 0, 0, time.UTC)
  endDate := time.Date(2025, time.March, 29, 0, 0, 0, 0, time.UTC)

  var dataToCopy []*v1.MyCustomDataset
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &dataToCopy,
    datasets.WithCollectionIDs(srcCollection.ID),
    datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  destCollection, err := client.Collections.GetOrCreate(ctx, dataset.ID, "OtherMeasurements")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  // copy the data to the other collection
  _, err = client.Datapoints.Ingest(ctx, destCollection.ID, &dataToCopy, false)
  if err != nil {
    log.Fatalf("Failed to ingest datapoints: %v", err)
  }

  // To verify it now contains 4 datapoints (2 we ingested already, and 2 we copied just now)
  updatedDestCollection, err := client.Collections.Get(ctx, dataset.ID, "OtherMeasurements")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }
  slog.Info("Updated collection", slog.String("collection", updatedDestCollection.String()))
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  OtherMeasurements: [2025-03-28T11:44:23.000 UTC, 2025-03-28T11:46:54.000 UTC] (4 data points)
  ```
</CodeGroup>

## Automatic batching

Tilebox automatically batches the ingestion requests for you, so you don't have to worry about the maximum request size.

## Idempotency

Tilebox will auto-generate datapoint IDs based on the data of all its fields - except for the auto-generated
`ingestion_time`, so ingesting the same data twice will result in the same ID being generated. By default, Tilebox
will silently skip any data points that are duplicates of existing ones in a collection. This behavior is especially
useful when implementing idempotent algorithms. That way, re-executions of certain ingestion tasks due to retries
or other reasons will never result in duplicate data points.

You can instead also request an error to be raised if any of the generated datapoint IDs already exist.
This can be done by setting the `allow_existing` parameter to `False`.

<CodeGroup>
  ```python Python theme={"system"}
  data = pd.DataFrame({
      "time": [
        "2025-03-28T11:45:19Z",
      ],
      "value": [45.16],
      "sensor": ["A"],
      "precise_time": [
        "2025-03-28T11:44:23.345761444Z",
      ],
      "sensor_history": [
        [-12.15, 13.45, -8.2, 16.5, 45.16],
      ],
  })

  # we already ingested the same data point previously
  collection.ingest(data, allow_existing=False)

  # we can still ingest it, by setting allow_existing=True
  # but the total number of datapoints will still be the same
  # as before in that case, since it already exists and therefore
  # will be skipped
  collection.ingest(data, allow_existing=True)  # no-op
  ```

  ```go Go theme={"system"}
  datapoints := []*v1.MyCustomDataset{
    v1.MyCustomDataset_builder{
      Time:          timestamppb.New(time.Date(2025, time.March, 28, 11, 45, 19, 0, time.UTC)),
      Value:         proto.Float64(45.16),
      Sensor:        proto.String("A"),
      PreciseTime:   timestamppb.New(time.Date(2025, time.March, 28, 11, 44, 23, 345761444, time.UTC)),
      SensorHistory: []float64{-12.15, 13.45, -8.2, 16.5, 45.16},
    }.Build(),
  }

  // we already ingested the same data point previously
  _, err = client.Datapoints.Ingest(ctx, collection.ID, &datapoints, false)
  if err != nil {
    log.Fatalf("Failed to ingest datapoints: %v", err)
  }

  // we can still ingest it, by setting allowExisting to true
  // but the total number of datapoints will still be the same
  // as before in that case, since it already exists and therefore
  // will be skipped
  _, err = client.Datapoints.Ingest(ctx, collection.ID, &datapoints, true) // no-op
  if err != nil {
    log.Fatalf("Failed to ingest datapoints: %v", err)
  }
  ```
</CodeGroup>

<CodeGroup>
  ```plaintext Output theme={"system"}
  ArgumentError: found existing datapoints with same id, refusing to ingest with "allow_existing=false"
  ```
</CodeGroup>

## Ingestion from common file formats

Through the usage of `xarray` and `pandas` you can also easily ingest existing datasets available in file
formats, such as CSV, [Parquet](https://parquet.apache.org/), [Feather](https://arrow.apache.org/docs/python/feather.html) and more.

Check out the [Ingestion from common file formats](/guides/datasets/ingest-format) guide for examples of how to achieve this.

## Geometries

Ingesting Geometries can traditionally be a bit tricky, especially when working with geometries that cross the antimeridian or cover a pole.
Tilebox is designed to take away most of the friction involved in this, but it's still recommended to follow the [best practices for handling geometries](/datasets/geometries).


# Tilebox Datasets
Source: https://docs.tilebox.com/datasets/introduction

A high-performance platform for structuring and querying satellite metadata, with curated open data catalogs and support for custom dataset collections.

Tilebox Datasets ingests and structures metadata for efficient querying, reducing data transfer and storage costs.

Create your own [Custom Datasets](/datasets/concepts/datasets) and easily set up a private, custom, strongly typed and highly available catalogue, or
explore any of the wide range of [available public open data datasets](/datasets/open-data) available on Tilebox.

Learn more about datasets by exploring the following sections:

<Columns>
  <Card title="Datasets" icon="database" href="/datasets/concepts/datasets">
    Learn what dataset types are available on Tilebox and how to create, list and access them.
  </Card>

  <Card title="Collections" icon="layer-group" href="/datasets/concepts/collections">
    Learn what collections are and how to access them.
  </Card>

  <Card title="Querying Data" icon="server" href="/datasets/query/querying-data">
    Find out how to access data from a collection for specific time intervals.
  </Card>

  <Card title="Ingesting Data" icon="up-from-bracket" href="/datasets/ingest">
    Learn how to ingest data into a collection.
  </Card>
</Columns>

<Note>
  For a quick reference to API methods or specific parameter meanings, [check out the complete datasets API Reference](/api-reference/python/tilebox.datasets/Client).
</Note>

## Terminology

Get familiar with some key terms when working with time series datasets.

<AccordionGroup>
  <Accordion title="Data points">
    Data points are the individual entities that form a dataset. Each data point has a set of required [fields](/datasets/types/timeseries) determined by the dataset type, and can have custom user-defined fields.
  </Accordion>

  <Accordion title="Datasets">
    Datasets act as containers for data points. All data points in a dataset share the same type and fields. Tilebox supports different types of datasets, currently those are [Timeseries](/datasets/types/timeseries) and [Spatio-temporal](/datasets/types/spatiotemporal) datasets.
  </Accordion>

  <Accordion title="Collections">
    Collections group data points within a dataset. They help represent logical groupings of data points that are often queried together.
  </Accordion>
</AccordionGroup>

## Creating a datasets client

Prerequisites

* You have installed the [python](/sdks/python/install) `tilebox-datasets` package or [go](/sdks/go/install) library.
* You have [created](/authentication) a Tilebox API key.

After meeting these prerequisites, you can create a client instance to interact with Tilebox Datasets.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client(token="YOUR_TILEBOX_API_KEY")
  ```

  ```go Go theme={"system"}
  import (
  	"github.com/tilebox/tilebox-go/datasets/v1"
  )

  client := datasets.NewClient(
  	datasets.WithAPIKey("YOUR_TILEBOX_API_KEY"),
  )
  ```
</CodeGroup>

You can also set the `TILEBOX_API_KEY` environment variable to your API key. You can then instantiate the client without passing the `token` argument. Python will automatically use this environment variable for authentication.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  # requires a TILEBOX_API_KEY environment variable
  client = Client()
  ```

  ```go Go theme={"system"}
  import (
  	"github.com/tilebox/tilebox-go/datasets/v1"
  )

  // requires a TILEBOX_API_KEY environment variable
  client := datasets.NewClient()
  ```
</CodeGroup>

<Tip>
  Tilebox datasets provide a standard synchronous API by default but also offers an [asynchronous client](/sdks/python/async) if needed.
</Tip>

### Exploring datasets

After creating a client instance, you can start exploring available datasets. A straightforward way to do this in an interactive environment is to [list all datasets](/api-reference/python/tilebox.datasets/Client.datasets) and use the autocomplete feature in your Jupyter notebook.

<CodeGroup>
  ```python Python theme={"system"}
  datasets = client.datasets()
  datasets. # trigger autocomplete here to view available datasets
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"github.com/tilebox/tilebox-go/datasets/v1"
  	"log"
  )

  func main() {
  	client := datasets.NewClient()

  	ctx := context.Background()
  	allDatasets, err := client.Datasets.List(ctx)
  	if err != nil {
  		log.Fatalf("Failed to list datasets: %v", err)
  	}

  	for _, dataset := range allDatasets {
  		log.Printf("Dataset: %s", dataset.Name)
  	}
  }
  ```
</CodeGroup>

<Tip>
  The Console also provides an [overview](https://console.tilebox.com/datasets/explorer) of all available datasets.
</Tip>

### Errors you might encounter

#### AuthenticationError

`AuthenticationError` occurs when the client fails to authenticate with the Tilebox API. This may happen if the provided API key is invalid or expired. A client instantiated with an invalid API key won't raise an error immediately, but an error will occur when making a request to the API.

<CodeGroup>
  ```python Python theme={"system"}
  client = Client(token="invalid-key") # runs without error
  datasets = client.datasets() # raises AuthenticationError
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"github.com/tilebox/tilebox-go/datasets/v1"
  	"log"
  )

  func main() {
  	// runs without error
  	client := datasets.NewClient(datasets.WithAPIKey("invalid-key"))

  	// returns an error
  	_, err := client.Datasets.List(context.Background())
  	if err != nil {
  		log.Fatalf("Failed to list datasets: %v", err)
  	}
  }
  ```
</CodeGroup>

## Next steps

<Columns>
  <Card title="Accessing datasets" icon="database" href="/datasets/concepts/datasets" />

  <Card title="Async support" icon="rotate" href="/sdks/python/async" />

  <Card title="Working with Xarray" icon="chart-bar" href="/sdks/python/xarray" />
</Columns>


# Open Data
Source: https://docs.tilebox.com/datasets/open-data

Browse the public satellite datasets available in Tilebox, ready to use for prototyping and building your applications before your own data is available.

On top of access to your own, private datasets, Tilebox provides access to a growing number of public datasets.
These datasets are available to all users of Tilebox and are a great way to get started and prototype your applications even before data from your own satellites is available.

<Note>
  If there is a dataset you would like to see in Tilebox, you can request it in the [Console open data page](https://console.tilebox.com/datasets/open-data).
</Note>

## Accessing Open Data through Tilebox

Accessing open datasets in Tilebox is straightforward and as easy as accessing your own datasets. Tilebox has already ingested the required metadata for each available dataset, so you can query, preview, and download the data seamlessly. This setup enables you to leverage performance optimizations and simplifies your workflows.

By using the [datasets API](/datasets), you can start prototyping your applications and workflows easily.

## Available datasets

<Tip>
  The Tilebox Console contains in-depth descriptions of each dataset. Check out the [Sentinel 5P Tropomi](https://console.tilebox.com/datasets/explorer/bb394de4-b47f-4069-bc4c-6e6a2c9f0641) documentation as an example.
</Tip>

### Copernicus Data Space

The [Copernicus Data Space](https://dataspace.copernicus.eu/) is an open ecosystem that provides free instant access to data and services from the Copernicus Sentinel missions.

Tilebox currently supports the following datasets from the Copernicus Data Space:

<AccordionGroup>
  <Accordion title="Sentinel 1">
    The Sentinel-1 mission is the European Radar Observatory for the Copernicus
    joint initiative of the European Commission (EC) and the European Space
    Agency (ESA). The Sentinel-1 mission includes C-band imaging operating in
    four exclusive imaging modes with different resolution (down to 5 m) and
    coverage (up to 400 km). It provides dual polarization capability, short
    revisit times and rapid product delivery.
  </Accordion>

  <Accordion title="Sentinel 2">
    Sentinel-2 is equipped with an optical instrument payload that samples 13
    spectral bands: four bands at 10 m, six bands at 20 m and three bands at 60
    m spatial resolution.
  </Accordion>

  <Accordion title="Sentinel 3">
    Sentinel-3 is equipped with multiple instruments whose data is available in Tilebox.

    `OLCI` (Ocean and Land Color Instrument) is an optical instrument used to provide
    data continuity for ENVISAT MERIS.

    `SLSTR` (Sea and Land Surface Temperature Radiometer) is a dual-view scanning
    temperature radiometer, which flies in low Earth orbit (800 - 830 km
    altitude).

    The `SRAL` (SAR Radar Altimeter) instrument comprises one nadir-looking antenna,
    and a central electronic chain composed of a Digital Processing Unit (DPU)
    and a Radio Frequency Unit (RFU).

    OLCI, in conjunction with the SLSTR instrument, provides the `SYN` products,
    providing continuity with SPOT VEGETATION.
  </Accordion>

  <Accordion title="Sentinel 5P">
    The primary goal of `TROPOMI` is to provide daily global observations of key
    atmospheric constituents related to monitoring and forecasting air quality,
    the ozone layer, and climate change.
  </Accordion>

  <Accordion title="Sentinel 6">
    The Sentinel-6 mission represents a groundbreaking advancement in Earth observation,
    providing invaluable insights for scientists, environmentalists, and stakeholders worldwide.
    At the heart of this mission is the cutting-edge radar altimeter instrument, designed to
    measure sea surface height and monitor key oceanographic parameters with unparalleled accuracy.
    The Sentinel-6 satellite collects precise data by employing advanced radar technology,
    allowing for a comprehensive understanding of sea level variations, ocean currents, and climate patterns.
  </Accordion>
</AccordionGroup>

### United States Geological Survey (USGS)

The [United States Geological Survey (USGS)](https://www.usgs.gov/) provides a wide range of Earth observation data, including [Landsat](https://science.nasa.gov/mission/landsat/) data, which are also available as open data through Tilebox.

<Frame>
  <img alt="USGS Landsat timeline" />
</Frame>

<AccordionGroup>
  <Accordion title="Landsat 1-3">
    Landsat-1/2/3 carry the Multispectral Scanner (MSS) instrument, which measures in the VIS and NIR portions of the spectrum. Its images have 79 m spatial resolution.
  </Accordion>

  <Accordion title="Landsat 4-5">
    Landsat-4/5 carry both the Multispectral Scanner (MSS) which measures in the VIS, NIR and SWIR
    portions of the spectrum in a 30m spatial resolution, and the Thematic Mapper (TM) instrument, which measures in the thermal-infrared band in a 120 m spatial resolution.
  </Accordion>

  <Accordion title="Landsat 7">
    Landsat-7 carries the Enhanced Thematic Mapper Plus (ETM+) instrument, which measures in the VIS, NIR and SWIR and thermal-infrared
    portions of the spectrum. Its images have 15-60 m spatial resolution.
  </Accordion>

  <Accordion title="Landsat 8-9">
    Landsat-8/9 are part of the long-running Landsat programme led by USGS and NASA and
    carry the Operational Land Imager (OLI) and the Thermal Infrared Sensor
    (TIRS). The Operational Land Imager (OLI), on board measures in
    the VIS, NIR and SWIR portions of the spectrum. Its images have 15 m
    panchromatic and 30 m multi-spectral spatial resolutions along a 185 km wide
    swath, covering wide areas of the Earth's landscape while providing
    high enough resolution to distinguish features like urban centres, farms,
    forests and other land uses. The entire Earth falls within view once every
    16 days due to Landsat-8/9's near-polar orbit. The Thermal Infra-Red Sensor (TIRS)
    instrument on board is a thermal imager operating in pushbroom
    mode with two Infra-Red channels: 10.8 µm and 12 µm with 100 m spatial
    resolution.
  </Accordion>
</AccordionGroup>

### Alaska Satellite Facility (ASF)

The [Alaska Satellite Facility (ASF)](https://asf.alaska.edu/) is a NASA-funded research center at the University of Alaska Fairbanks. ASF supports a wide variety of research and applications using synthetic aperture radar (SAR) and related remote sensing technologies.

Tilebox currently supports the following datasets from the Alaska Satellite Facility:

<AccordionGroup>
  <Accordion title="ERS SAR">
    European Remote Sensing Satellite (ERS) Synthetic Aperture Radar (SAR) Granules
  </Accordion>
</AccordionGroup>

### Umbra Space

[Umbra](https://umbra.space/) satellites provide up to 16 cm resolution Synthetic Aperture Radar (SAR) imagery from space. The Umbra Open Data Program (ODP) features over twenty diverse time-series locations that are frequently updated, allowing users to explore SAR's capabilities.

Tilebox currently supports the following datasets from Umbra Space:

<AccordionGroup>
  <Accordion title="Umbra SAR">
    Time-series SAR data provided as Opendata by Umbra Space.
  </Accordion>
</AccordionGroup>

### Satellogic

[Satellogic](https://satellogic.com/) offers the EarthView dataset, which includes high-resolution satellite images captured over all continents. The dataset is available through Tilebox as a spatio-temporal dataset.

<AccordionGroup>
  <Accordion title="Satellogic EarthView">
    High resolution satellite images provided as Opendata by Satellogic. The dataset provides Top-of-Atmosphere (TOA) reflectance values across four spectral bands (Red, Green, Blue, Near-Infrared) at a Ground Sample Distance (GSD) of 1 meter, accompanied by comprehensive metadata such as off-nadir angles, sun elevation, and other pertinent details. Users should note that due to an artifact in region delineation, a small number of regions present overlaps.
  </Accordion>
</AccordionGroup>

### Wyvern

[Wyvern](https://wyvern.space/) offers a constellation of satellites providing high-resolution, hyperspectral imagery. They offer products from their [Dragonette constellation](https://wyvern.space/our-products/generation-one-hyperspectral-satellites/) as open data, which is also available through Tilebox as a spatio-temporal dataset.

<AccordionGroup>
  <Accordion title="Wyvern Dragonette">
    High resolution, hyperspectral imagery provided as Opendata by Wyvern.
  </Accordion>
</AccordionGroup>


# Querying individual datapoints by ID
Source: https://docs.tilebox.com/datasets/query/filter-by-id

Look up specific datapoints by ID across one or more dataset collections, without needing to construct and execute a broader query.

If you already know the ID of the datapoint you want to query, you can fetch it directly without needing to construct and execute a broader query.
You can query a datapoint ID either in only specific collection of a dataset, a selected set of collections of a dataset, or from all collections of a dataset at once.

<CodeGroup>
  ```python Python theme={"system"}
  datapoint_id = "0197a491-1520-102f-48f4-f087d6ef8603"

  # query in all collections of a dataset
  datapoint = dataset.find(datapoint_id)

  # query in selected collections of a dataset
  datapoint = dataset.find(
      datapoint_id,
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A"],
  )

  # query in a single collection
  datapoint = dataset.collection("S2A_S2MSI2A").find(
      datapoint_id
  )

  print(datapoint)
  ```

  ```go Go theme={"system"}
  datapointID := uuid.MustParse("0197a491-1520-102f-48f4-f087d6ef8603")

  // query in all collections of a dataset
  var datapoint v1.Sentinel2Msi
  err = client.Datapoints.GetInto(ctx,
    dataset.ID,
    datapointID,
    &datapoint,
  )
  if err != nil {
    log.Fatalf("Failed to query datapoint: %v", err)
  }

  collectionA, err := client.Collections.Get(ctx, dataset.ID, "S2A_S2MSI2A")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  collectionB, err := client.Collections.Get(ctx, dataset.ID, "S2B_S2MSI2A")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  // query in selected collections of a dataset
  var datapointFromSelectedCollections v1.Sentinel2Msi
  err = client.Datapoints.GetInto(ctx,
    dataset.ID,
    datapointID,
    &datapointFromSelectedCollections,
    datasets.WithCollectionIDs(collectionA.ID, collectionB.ID),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoint: %v", err)
  }

  // query in a single collection
  var datapointFromSingleCollection v1.Sentinel2Msi
  err = client.Datapoints.GetInto(ctx,
    dataset.ID,
    datapointID,
    &datapointFromSingleCollection,
    datasets.WithCollectionIDs(collectionA.ID),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoint: %v", err)
  }

  fmt.Println(protojson.Format(&datapoint))
  ```
</CodeGroup>

Output

<CodeGroup>
  ```plaintext Python theme={"system"}
  <xarray.Dataset> Size: 443B
  Dimensions:                ()
  Coordinates:
      time                   datetime64[ns] 8B 2025-06-25T00:51:01.024000
  Data variables: (12/23)
      id                     <U36 144B '0197a491-1520-102f-48f4-f087d6ef8603'
      ingestion_time         datetime64[ns] 8B 2025-06-25T05:33:11.104000
      geometry               object 8B POLYGON ((156.090908 48.709728, 155.7188...
      granule_name           object 8B 'S2A_MSIL2A_20250625T005101_N0511_R045_T...
      processing_level       uint8 1B 5
      product_type           object 8B 'S2MSI2A'
      ...                     ...
      thumbnail              object 8B 'https://catalogue.dataspace.copernicus....
      cloud_cover            float64 8B 87.38
      resolution             int64 8B 0
      flight_direction       uint8 1B 2
      acquisition_mode       uint8 1B 20
      mission_take_id        object 8B 'GS2A_20250625T005101_052266_N05.11'
  ```

  ```plaintext Go theme={"system"}
  {
    "time": "2025-06-25T00:51:01.024Z",
    "id": {
      "uuid": "AZekkRUgEC9I9PCH1u+GAw=="
    },
    "ingestionTime": "2025-06-25T05:33:11.104Z",
    "geometry": {
      "wkb": "AQMAACDmEAAAAQAAAAoAAAA83uS36IJjQBqH+l3YWkhA+ptQiAB3Y0CmYmNeR1xIQGfROxVwdWNA58OzBBnlR0DAB69d2nZjQIQqNXug8UdABhN/FPV4Y0AwLH++LQRIQCxEh8ARe2NAhEpcx7gWSEC+UMB2MH1jQCZuFcRAKUhAxVkRNVF/Y0Cs4LchxjtIQE6AYflzgWNABAMIH0pOSEA83uS36IJjQBqH+l3YWkhA"
    },
    "granuleName": "S2A_MSIL2A_20250625T005101_N0511_R045_T56UQU_20250625T033500.SAFE",
    "processingLevel": "PROCESSING_LEVEL_L2A",
    "productType": "S2MSI2A",
    "copernicusId": {
      "uuid": "+gUdtyZrQnapFVrQjnUKUw=="
    },
    "platform": "S2A",
    "orbitNumber": "52266",
    "relativeOrbitNumber": "45",
    "processingBaseline": 5.11,
    "stopTime": "2025-06-25T00:51:01.024Z",
    "centroid": {
      "wkb": "AQEAACDmEAAAmRmrzXB6Y0ATN84DLTRIQA=="
    },
    "published": "2025-06-25T04:29:22.500212Z",
    "updated": "2025-06-25T04:29:22.500212Z",
    "location": "/eodata/Sentinel-2/MSI/L2A/2025/06/25/S2A_MSIL2A_20250625T005101_N0511_R045_T56UQU_20250625T033500.SAFE",
    "fileSize": "118828098",
    "thumbnail": "https://catalogue.dataspace.copernicus.eu/get-object?path=/Sentinel-2/MSI/L2A/2025/06/25/S2A_MSIL2A_20250625T005101_N0511_R045_T56UQU_20250625T033500.SAFE/S2A_MSIL2A_20250625T005101_N0511_R045_T56UQU_20250625T033500-ql.jpg",
    "cloudCover": 87.384349,
    "resolution": "0",
    "flightDirection": "FLIGHT_DIRECTION_DESCENDING",
    "acquisitionMode": "ACQUISITION_MODE_NOBS",
    "missionTakeId": "GS2A_20250625T005101_052266_N05.11"
  }
  ```
</CodeGroup>

<Tip>
  You can also set the `skip_data` parameter when calling `find` to query only the required fields of the data point, same as for `query`.
</Tip>

## Checking if a datapoint exists

`find` returns an error if the specified datapoint does not exist. You can use this to check if a datapoint exists or not.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets.sync.dataset import NotFoundError

  datapoint_id = "0197a47f-a830-1160-6df5-61ac723dae17"  # doesn't exist

  try:
      dataset.find(datapoint_id)
      exists = True
  except NotFoundError:
      exists = False
  ```

  ```go Go theme={"system"}
  datapointID := uuid.MustParse("0197a47f-a830-1160-6df5-61ac723dae17")

  exists := true
  var datapoint v1.Sentinel2Msi
  err = client.Datapoints.GetInto(ctx,
    dataset.ID,
    datapointID,
    &datapoint,
  )
  if err != nil {
    if connect.CodeOf(err) == connect.CodeNotFound {
        exists = false
    } else {
      log.Fatalf("Failed to query datapoint: %v", err)
    }
  }
  ```
</CodeGroup>


# Filtering by a location
Source: https://docs.tilebox.com/datasets/query/filter-by-location

Narrow down your query results to only include datapoints within a specific geographic area of interest by providing a geometry for the target region.

<Tip>
  Check out the [best practices for handling geometries](/datasets/geometries) to learn more about the different aspects to consider when working with geometries, including antimeridian crossings and pole coverings.
</Tip>

When querying, you can specify arbitrary geometries as an area of interest. Tilebox currently supports `Polygon` and `MultiPolygon` geometries as query filters.

## Filtering by Area of Interest

To filter by an area of interest, use a `Polygon` or `MultiPolygon` geometry as the spatial extent parameter.

Here is how to query Sentinel-2 `L2A` data over Colorado for a specific day in April 2025.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import Polygon
  from tilebox.datasets import Client

  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  client = Client()
  sentinel2_msi = client.dataset("open_data.copernicus.sentinel2_msi")
  data = sentinel2_msi.query(
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A", "S2C_S2MSI2A"],
      temporal_extent=("2025-04-02", "2025-04-03"),
      spatial_extent=area,
  )
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 2, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 4, 3, 0, 0, 0, 0, time.UTC)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  ctx := context.Background()
  client := datasets.NewClient()

  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")
  if err != nil {
      log.Fatalf("Failed to get dataset: %v", err)
  }

  collection, err := client.Collections.Get(ctx, dataset.ID, "S2A_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  collectionB, err := client.Collections.Get(ctx, dataset.ID, "S2B_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  collectionC, err := client.Collections.Get(ctx, dataset.ID, "S2C_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithCollectionIDs(collection.ID, collectionB.ID, collectionC.ID),
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
      datasets.WithSpatialExtent(area),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

## Intersection mode

By default, queries return all datapoints that intersect with the specified geometry. You can alter this behavior to return only datapoints fully contained within the geometry. Tilebox supports this by allowing you to specify a mode for the spatial filter.

<Columns>
  <Frame>
    <img alt="Query results with intersects mode" />

    <img alt="Query results with intersects mode" />
  </Frame>

  <Frame>
    <img alt="Query results with contains mode" />

    <img alt="Query results with contains mode" />
  </Frame>
</Columns>

### Intersects

The `intersects` mode is the default for spatial queries. It matches all datapoints whose geometries intersect with the query geometry.

<CodeGroup>
  ```python Python theme={"system"}
  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  data = dataset.query(
      temporal_extent=("2025-04-02", "2025-04-03"),
      # intersects is the default, so can also be omitted entirely
      spatial_extent={"geometry": area, "mode": "intersects"},
  )
  print(f"There are {data.sizes['time']} Sentinel-2A granules intersecting the area of Colorado on April 2nd, 2025")
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 2, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 4, 3, 0, 0, 0, 0, time.UTC)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  var datapoints []*examplesv1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
      datasets.WithSpatialExtentFilter(&query.SpatialFilter{
          Geometry: area,
          // intersects is the default, so can also be omitted entirely
          Mode:     datasetsv1.SpatialFilterMode_SPATIAL_FILTER_MODE_INTERSECTS,
      }),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
There are 27 Sentinel-2A granules intersecting the area of Colorado on April 2nd, 2025
```

### Contains

The `contains` mode matches all datapoints whose geometries are fully contained within the query geometry.

<CodeGroup>
  ```python Python theme={"system"}
  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  data = collection.query(
      temporal_extent=("2025-04-01", "2025-05-02"),
      spatial_extent={"geometry": area, "mode": "contains"},
  )
  print(f"There are {data.sizes['time']} Sentinel-2A granules fully contained within the area of Colorado on April 2nd, 2025")
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 2, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 5, 2, 0, 0, 0, 0, time.UTC)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  var datapoints []*examplesv1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithCollectionIDs(collection.ID),
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
      datasets.WithSpatialExtentFilter(&query.SpatialFilter{
          Geometry: area,
          Mode:     datasetsv1.SpatialFilterMode_SPATIAL_FILTER_MODE_CONTAINS,
      }),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
There are 16 Sentinel-2A granules fully contained within the area of Colorado on April 2nd, 2025
```

## Antimeridian Crossings

In many applications, geometries that cross the antimeridian cause issues. Since such geometries are common in satellite
data, Tilebox does take extra care to handle them out of the box correctly, by building the necessary internal spatial
index structures in a way that correctly handles antimeridian crossings and pole coverings.

To get accurate results also at query time, it's recommend to use the `spherical` [coordinate reference system](#coordinate-reference-system)
for querying (which is the default), as it correctly handles the non-linearity introduced by the antimeridian in `cartesian` space.

<Tip>
  Even if you stick to the `spherical` coordinate reference system when querying, it's still recommended to follow the
  [best practices for handling geometries](/datasets/geometries). In doing so,
  you can ensure that no geometry related issues will arise even when interfacing with other libraries and tools that may not properly
  support non-linearities in geometries.
</Tip>

## Coordinate reference system

Geometry intersection and containment checks can either be performed in a [3D spherical coordinate system](https://en.wikipedia.org/wiki/Spherical_coordinate_system)
or in a standard 2D cartesian `lat/lon` coordinate system.

<Columns>
  <Frame>
    <img alt="Spherical coordinate reference system" />

    <img alt="Spherical coordinate reference system" />
  </Frame>

  <Frame>
    <img alt="Cartesian coordinate reference system" />

    <img alt="Cartesian coordinate reference system" />
  </Frame>
</Columns>

### Spherical

The `spherical` coordinate reference system is the default and recommended choice. It correctly handles antimeridian
crossings and is the most robust option, regardless of how datapoint geometries are cut along the antimeridian.

<Tip>
  To learn more about antimeridian crossings and how to handle them correctly, check out the [Antimeridian Crossings section](#antimeridian-crossings) above.
</Tip>

When querying with the `spherical` coordinate reference system, Tilebox automatically converts all geometries to
their `x, y, z` coordinates on the unit sphere and performs the intersection and containment checks in 3D.

<CodeGroup>
  ```python Python theme={"system"}
  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  data = dataset.query(
      temporal_extent=("2025-04-01", "2025-05-02"),
      # spherical is the default, so can also be omitted entirely
      spatial_extent={"geometry": area, "coordinate_system": "spherical"},
  )
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 2, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 5, 2, 0, 0, 0, 0, time.UTC)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  var datapoints []*examplesv1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
      datasets.WithSpatialExtentFilter(&query.SpatialFilter{
          Geometry: area,
          // spherical is the default, so can also be omitted entirely
          CoordinateSystem: datasetsv1.SpatialCoordinateSystem_SPATIAL_COORDINATE_SYSTEM_SPHERICAL,
      }),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

### Cartesian

Tilebox can also be configured to use a standard 2D cartesian `lat/lon` coordinate system for geometry intersection and containment checks.
This is done by specifying the `cartesian` coordinate reference system when querying.

<CodeGroup>
  ```python Python theme={"system"}
  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  data = dataset.query(
      temporal_extent=("2025-04-01", "2025-05-02"),
      spatial_extent={"geometry": area, "coordinate_system": "cartesian"},
  )
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 2, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 5, 2, 0, 0, 0, 0, time.UTC)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  var datapoints []*examplesv1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
      datasets.WithSpatialExtentFilter(&query.SpatialFilter{
          Geometry: area,
          CoordinateSystem: datasetsv1.SpatialCoordinateSystem_SPATIAL_COORDINATE_SYSTEM_CARTESIAN,
      }),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

<Note>
  When using the `cartesian` coordinate system, antimeridian crossings may cause issues if datapoint geometries
  or the query geometry do not properly respect the antimeridian cut. Check out the [best practices](/datasets/geometries#best-practices)
  for handling geometries to learn more about how to avoid such issues.
</Note>


# Querying by temporal extent
Source: https://docs.tilebox.com/datasets/query/filter-by-time

Retrieve datapoints that fall within a given time period, with support for precise time intervals, open-ended time ranges, and exact timestamp lookups.

Both [Timeseries](/datasets/types/timeseries) and [Spatio-temporal](/datasets/types/spatiotemporal) datasets support efficient time-based queries.

## Time interval queries

To query data for a specific time interval, use a `tuple` in the form `(start, end)` as the `temporal_extent` parameter. Both `start` and `end` must be [TimeScalars](#time-scalar-queries), which can be `datetime` objects or strings in ISO 8601 format.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  sentinel2_msi = client.dataset("open_data.copernicus.sentinel2_msi")
  data = sentinel2_msi.query(
    collections=["S2A_S2MSI2A", "S2B_S2MSI2A", "S2C_S2MSI2A"],
    temporal_extent=("2025-05-01", "2025-06-01"),
    show_progress=True,
  )

  print(f"Queried {data.sizes['time']} data points.")
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, time.May, 1, 0, 0, 0, 0, time.UTC)
  endDate := time.Date(2025, time.June, 1, 0, 0, 0, 0, time.UTC)
  interval := query.NewTimeInterval(startDate, endDate)

  ctx := context.Background()
  client := datasets.NewClient()

  dataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")
  if err != nil {
    log.Fatalf("Failed to get dataset: %v", err)
  }

  collection, err := client.Collections.Get(ctx, dataset.ID, "S2A_S2MSI2A")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  collectionB, err := client.Collections.Get(ctx, dataset.ID, "S2B_S2MSI2A")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  collectionC, err := client.Collections.Get(ctx, dataset.ID, "S2C_S2MSI2A")
  if err != nil {
    log.Fatalf("Failed to get collection: %v", err)
  }

  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithCollectionIDs(collection.ID, collectionB.ID, collectionC.ID),
    datasets.WithTemporalExtent(interval),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  log.Printf("Queried %d datapoints", len(datapoints))
  ```
</CodeGroup>

```plaintext Output theme={"system"}
Queried 481079 data points.
```

<Note>
  The `show_progress` parameter is optional and can be used to display a [tqdm](https://tqdm.github.io/) progress bar while loading data.
</Note>

### Endpoint inclusivity

A time interval specified as a tuple is interpreted as a half-closed interval. This means the start time is inclusive, and the end time is exclusive.
For instance, using an end time of `2025-06-01` includes data points up to `2025-05-31 23:59:59.999`, but excludes those from `2025-06-01 00:00:00.000`.
This behavior mimics the Python `range` function and is useful for chaining time intervals.

<CodeGroup>
  ```python Python theme={"system"}
  import xarray as xr

  data = []
  for month in [4, 5, 6]:
      interval = (f"2025-{month}-01", f"2025-{month+1}-01")
      data.append(collection.query(temporal_extent=interval, show_progress=True))

  # Concatenate the data into a single dataset, which is equivalent
  # to the result of the single request in the code example above.
  data = xr.concat(data, dim="time")
  ```

  ```go Go theme={"system"}
  var datapoints []*v1.Sentinel2Msi

  for month := 4; month <= 6; month++ {
    startDate := time.Date(2025, month, 1, 0, 0, 0, 0, time.UTC)
    endDate := time.Date(2025, month + 1, 1, 0, 0, 0, 0, time.UTC)

    var partialDatapoints []*v1.Sentinel2Msi
    err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &partialDatapoints,
      datasets.WithCollectionIDs(collection.ID),
      datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)),
    )
    if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
    }

    // Concatenate the data into a single dataset, which is equivalent
    // to the result of the single request in the code example above.
    datapoints = append(datapoints, partialDatapoints...)
  }
  ```
</CodeGroup>

Above example demonstrates how to split a large time interval into smaller chunks while loading data in separate requests. Typically, this is not necessary as the datasets client auto-paginates large intervals.

### Manual endpoint inclusivity

For greater control over inclusivity of start and end times, you can explicitly specify a `TimeInterval`. This way you can specify both the `start` and `end` times, as well as their inclusivity. Here's an example of creating equivalent `TimeInterval` objects in two different ways.

<CodeGroup>
  ```python Python theme={"system"}
  from datetime import datetime
  from tilebox.datasets.query import TimeInterval

  interval1 = TimeInterval(
      datetime(2021, 1, 1), datetime(2023, 1, 1),
      end_inclusive=False
  )
  interval2 = TimeInterval(
      # python datetime granularity is in milliseconds
      datetime(2021, 1, 1), datetime(2022, 12, 31, 23, 59, 59, 999999),
      end_inclusive=True
  )

  print("Inclusivity is indicated by interval notation: ( and [")
  print(interval1)
  print(interval2)
  print(f"They are equivalent: {interval1 == interval2}")
  print(interval2.to_half_open())

  # Query data for a time interval
  data = collection.query(temporal_extent=interval1, show_progress=True)
  ```

  ```go Go theme={"system"}
  interval1 := query.TimeInterval{
    Start:        time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC),
    End:          time.Date(2023, time.January, 1, 0, 0, 0, 0, time.UTC),
    EndInclusive: false,
  }

  interval2 := query.TimeInterval{
    Start:        time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC),
    // the granularity of time.Time in Go is nanoseconds
    End:          time.Date(2022, time.December, 31, 23, 59, 59, 999999999, time.UTC),
    EndInclusive: true,
  }

  log.Println("Inclusivity is indicated by interval notation: ( and [")
  log.Println(interval1.String())
  log.Println(interval2.String())
  log.Println("They are equivalent:", interval1.Equal(&interval2))
  log.Println(interval2.ToHalfOpen().String())

  // Query data for a time interval
  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithCollectionIDs(collection.ID),
    datasets.WithTemporalExtent(interval1),
  )
  ```
</CodeGroup>

```plaintext Output theme={"system"}
Inclusivity is indicated by interval notation: ( and [
[2021-01-01T00:00:00.000 UTC, 2023-01-01T00:00:00.000 UTC)
[2021-01-01T00:00:00.000 UTC, 2022-12-31T23:59:59.999 UTC]
They are equivalent: True
[2021-01-01T00:00:00.000 UTC, 2023-01-01T00:00:00.000 UTC)
```

## Time scalar queries

You can query all datapoints linked to a specific timestamp by specifying a `TimeScalar` as the time query argument. A `TimeScalar` can be a `datetime` object or a string in ISO 8601 format.

<Tip>
  Tilebox uses millisecond precision for time indexing datapoints. Thus, querying a specific time scalar, is equivalent to a time interval query of length 1 millisecond.
</Tip>

Here's how to query a data point at a specific millisecond from a [dataset](/datasets/concepts/datasets) or [collection](/datasets/concepts/collections).

<CodeGroup>
  ```python Python theme={"system"}
  data = sentinel2_msi.query(temporal_extent="2025-06-15T02:31:41.024")
  print(f"Queried {data.sizes['time']} data points.")
  first_timestamp = data.time[0].dt.strftime("%Y-%m-%dT%H:%M:%S.%f").item()
  print("First datapoint time:", first_timestamp)
  ```

  ```go Go theme={"system"}
  temporalExtent := query.NewPointInTime(time.Date(2025, time.June, 15, 2, 31, 41, 024000000, time.UTC))

  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithTemporalExtent(temporalExtent),
  )

  log.Printf("Queried %d datapoints", len(datapoints))
  log.Printf("First datapoint time: %s", datapoints[0].GetTime().AsTime())
  ```
</CodeGroup>

```plaintext Output theme={"system"}
Queried 714 datapoints
First datapoint time: 2025-06-15T02:31:41.024 +0000 UTC
```

<Tip>
  A collection may contain multiple datapoints for the same millisecond, so multiple data points may be returned. If you want to fetch only a single data point, [query the collection by id](#loading-a-data-point-by-id) instead.
</Tip>

## Timezone handling

All `TimeScalars` specified as a string are treated as UTC if they do not include a timezone suffix. If you want to query data for a specific time or time range
in another timezone, it's recommended to a type that includes timezone information. Tilebox will automatically convert such objects to `UTC` in order to send the right query requests.
All outputs will always contain UTC timestamps, which will need to be converted again to a different timezone if required.

<CodeGroup>
  ```python Python theme={"system"}
  from datetime import datetime
  import pytz

  # Tokyo has a UTC+9 hours offset, so this is the same as
  # 2017-01-01 02:45:25.679 UTC
  tokyo_time = pytz.timezone('Asia/Tokyo').localize(
      datetime(2021, 1, 1, 11, 45, 25, 679000)
  )
  print(tokyo_time)
  data = collection.query(temporal_extent=tokyo_time)
  print(data)
  ```

  ```go Go theme={"system"}
  // Tokyo has a UTC+9 hours offset, so this is the same as
  // 2017-01-01 02:45:25.679 UTC
  location, _ := time.LoadLocation("Asia/Tokyo")
  tokyoTime := query.NewPointInTime(time.Date(2021, 1, 1, 11, 45, 25, 679000000, location))
  log.Println(tokyoTime)

  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithCollectionIDs(collection.ID),
    datasets.WithTemporalExtent(tokyoTime),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  log.Printf("Queried %d datapoints", len(datapoints))
  // time is in UTC since API always returns UTC timestamps
  log.Printf("First datapoint time: %s", datapoints[0].GetTime().AsTime())
  ```
</CodeGroup>

Output

<CodeGroup>
  ```plaintext Python theme={"system"}
  2021-01-01 11:45:25.679000+09:00
  <xarray.Dataset> Size: 725B
  Dimensions:                (time: 1, latlon: 2)
  Coordinates:
      ingestion_time         (time) datetime64[ns] 8B 2024-06-21T11:03:33.852435
      id                     (time) <U36 144B '015957ea-d82f-e454-34ab-a87603ee...
    * time                   (time) datetime64[ns] 8B 2017-01-01T02:45:25.679000
    * latlon                 (latlon) <U9 72B 'latitude' 'longitude'
  Data variables:
      ...
  ```

  ```plaintext Go theme={"system"}
  [2021-01-01 11:45:25.679 +0900 JST, 2021-01-01 11:45:25.679 +0900 JST]
  Queried 1 datapoints
  First datapoint time: 2021-01-01 02:45:25.679 +0000 UTC
  ```
</CodeGroup>


# Querying data
Source: https://docs.tilebox.com/datasets/query/querying-data

Access and filter data stored in your datasets using time-based and spatial queries, with built-in support for pagination and progress tracking.

Tilebox offers a powerful and flexible querying API to access and filter data from your datasets. When querying, you can
[filter by time](/datasets/query/filter-by-time) and for [Spatio-temporal datasets](/datasets/types/spatiotemporal) optionally also [filter by a location in the form of a geometry](/datasets/query/filter-by-location).

## Running a query

You can query data from either a specific collection of a dataset, from a selected set of collections of a dataset, or from all collections of a dataset at once.
Below is a simple example showcasing those options by querying Sentinel-2 data for April 2025 over the state of Colorado.

<CodeGroup>
  ```python Python theme={"system"}
  from shapely import Polygon
  from tilebox.datasets import Client

  area = Polygon(  # area roughly covering the state of Colorado
      ((-109.05, 41.00), (-109.045, 37.0), (-102.05, 37.0), (-102.05, 41.00), (-109.05, 41.00)),
  )

  client = Client()
  sentinel2_msi = client.dataset("open_data.copernicus.sentinel2_msi")

  # query data from a specific collection
  collection = sentinel2_msi.collection("S2A_S2MSI2A")
  data = collection.query(
      temporal_extent=("2025-04-01", "2025-05-01"),
      spatial_extent=area,
      show_progress=True,
  )

  # query data from a selected set of collections
  data = sentinel2_msi.query(
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A", "S2C_S2MSI2A"],
      temporal_extent=("2025-04-01", "2025-05-01"),
      spatial_extent=area,
      show_progress=True,
  )

  # query data from all collections in the dataset
  data = sentinel2_msi.query(
      # omit the collections argument to query all collections
      temporal_extent=("2025-04-01", "2025-05-01"),
      spatial_extent=area,
      show_progress=True,
  )
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2025, 4, 1, 0, 0, 0, 0, time.UTC) 
  endDate := time.Date(2025, 5, 1, 0, 0, 0, 0, time.UTC)
  timeInterval := query.NewTimeInterval(startDate, endDate)
  area := orb.Polygon{  // area roughly covering the state of Colorado
      {{-109.05, 41.00}, {-109.045, 37.0},  {-102.05, 37.0}, {-102.05, 41.00}, {-109.05, 41.00}},
  }

  collectionA, err := client.Collections.Get(ctx, dataset.ID, "S2A_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  collectionB, err := client.Collections.Get(ctx, dataset.ID, "S2B_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  collectionC, err := client.Collections.Get(ctx, dataset.ID, "S2C_S2MSI2A")
  if err != nil {
      log.Fatalf("Failed to get collection: %v", err)
  }

  // query data from a specific collection
  var fromCollection []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &fromCollection,
      datasets.WithCollectionIDs(collectionA.ID),
      datasets.WithTemporalExtent(timeInterval),
      datasets.WithSpatialExtent(area),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }

  // query data from a selected set of collections
  var datapoints []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapoints,
      datasets.WithCollectionIDs(collectionA.ID, collectionB.ID, collectionC.ID),
      datasets.WithTemporalExtent(timeInterval),
      datasets.WithSpatialExtent(area),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }

  // query data from all collections in the dataset
  var datapointsFromAllCollections []*v1.Sentinel2Msi
  err = client.Datapoints.QueryInto(ctx,
      dataset.ID,
      &datapointsFromAllCollections,
      datasets.WithTemporalExtent(timeInterval),
      datasets.WithSpatialExtent(area),
  )
  if err != nil {
      log.Fatalf("Failed to query datapoints: %v", err)
  }
  ```
</CodeGroup>

To learn more about how to specify filters to narrow down the query results, check out the following sections about filtering by
time, by geometry or by datapoint ID.

<Columns>
  <Card title="Filter by time" icon="timeline" href="/datasets/query/filter-by-time">
    Learn how to run queries for datapoints within a given temporal extent.
  </Card>

  <Card title="Filter by geometry" icon="globe-pointer" href="/datasets/query/filter-by-location">
    Learn how to filter your query results by a certain geographical extent.
  </Card>

  <Card title="Filter by Datapoint ID" icon="fingerprint" href="/datasets/query/filter-by-id">
    Find individual datapoints by their unique ID.
  </Card>
</Columns>

## Automatic pagination

Querying large datasets can result in a large number of data points. For those cases Tilebox
automatically handles pagination for you by sending paginated requests to the server.

<Tip>
  When using the python SDK in an interactive notebook environment, you can additionally also display a
  progress bar to keep track of the progress of the query by setting the `show_progress` parameter to `True`.
</Tip>

## Skipping data fields

Sometimes, only the ID or timestamp associated with a datapoint is required. In those cases, you can speed up
querying by skipping downloading of all dataset fields except of the `time`, the `id` and the `ingestion_time` by setting the `skip_data` parameter to `True`.

For example, when checking how many datapoints exist in a given time interval, you can use `skip_data=True` to avoid loading the data fields.
This works the same way on both collection-level and dataset-level queries.

<CodeGroup>
  ```python Python theme={"system"}
  interval = ("2023-01-01", "2023-02-01")
  data = collection.query(temporal_extent=interval, skip_data=True)
  print(f"Found {data.sizes['time']} data points.")
  ```

  ```go Go theme={"system"}
  startDate := time.Date(2023, time.January, 1, 0, 0, 0, 0, time.UTC)
  endDate := time.Date(2023, time.February, 1, 0, 0, 0, 0, time.UTC)
  interval := query.NewTimeInterval(startDate, endDate)

  var datapoints []*v1.Sentinel1Sar
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithCollectionIDs(collection.ID),
    datasets.WithTemporalExtent(interval),
    datasets.WithSkipData(),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  log.Printf("Queried %d datapoints", len(datapoints))
  log.Printf("First datapoint time: %s", datapoints[0].GetTime().AsTime())
  ```
</CodeGroup>

Output

<CodeGroup>
  ```plaintext Python theme={"system"}
  <xarray.Dataset> Size: 160B
  Dimensions:         (time: 1)
  Coordinates:
      ingestion_time  (time) datetime64[ns] 8B 2024-08-01T08:53:08.450499
      id              (time) <U36 144B '01910b3c-8552-7671-3345-b902cc0813f3'
    * time            (time) datetime64[ns] 8B 2024-08-01T00:00:01.362000
  Data variables:
      *empty*
  ```

  ```plaintext Go theme={"system"}
  Queried 1 datapoints
  First datapoint time: 2024-08-01 00:00:01.362 +0000 UTC
  ```
</CodeGroup>

## Empty response

Query will not raise an error if no data points are found for the specified query, instead an empty result is returned.

In python, this is an empty `xarray.Dataset` object. In Go, an empty slice of datapoints. To check for an empty response, you can coerce the result to a boolean
or check the length of the slice.

<CodeGroup>
  ```python Python theme={"system"}
  timestamp_with_no_data_points = "1997-02-06 10:21:00:000"
  data = collection.query(temporal_extent=timestamp_with_no_data_points)
  if not data:
      print("No data points found")
  ```

  ```go Go theme={"system"}
  timeWithNoDatapoints := query.NewPointInTime(time.Date(1997, time.February, 6, 10, 21, 0, 0, time.UTC))

  var datapoints []*v1.Sentinel1Sar
  err = client.Datapoints.QueryInto(ctx,
    dataset.ID,
    &datapoints,
    datasets.WithCollectionIDs(collection.ID),
    datasets.WithTemporalExtent(timeWithNoDatapoints),
  )
  if err != nil {
    log.Fatalf("Failed to query datapoints: %v", err)
  }

  if len(datapoints) == 0 {
    log.Println("No data points found")
  }
  ```
</CodeGroup>

Output

<CodeGroup>
  ```plaintext Python theme={"system"}
  No data points found
  ```

  ```plaintext Go theme={"system"}
  No data points found
  ```
</CodeGroup>


# Storage Clients
Source: https://docs.tilebox.com/datasets/storage/clients

Configure and use storage clients in the Tilebox Python SDK to access satellite data products from public providers and local file systems.

Tilebox does not host the actual open data satellite products but instead relies on publicly accessible storage providers for data access.
Tilebox ingests available metadata as [datasets](/datasets/concepts/datasets) to enable high performance querying and structured access of the data as [xarray.Dataset](/sdks/python/xarray).

Tilebox also supports a local file system storage client. This is useful when your data is already available on a local disk, a mounted network share, or a synced folder such as Dropbox or Google Drive.

Below is a list of the storage providers currently supported by Tilebox.

<Note>
  This feature is only available in the Python SDK.
</Note>

## Local File System (including Dropbox-synced folders)

Use `LocalFileSystemStorageClient` when your dataset datapoints already reference files on a local or mounted path.

This client does not download remote data. Instead, it resolves and validates local paths using each datapoint's `location` field.

```python Python highlight={4, 11, 21} theme={"system"}
from pathlib import Path

from tilebox.datasets import Client
from tilebox.storage import LocalFileSystemStorageClient

# Creating clients
client = Client()
storage_client = LocalFileSystemStorageClient(root=Path("/Volumes/data"))

# Querying a dataset that stores file locations
dataset = client.dataset("my_org.local.imagery")
collection = dataset.collection("ORTHO")
data = collection.query(temporal_extent=("2025-01-01", "2025-01-02"), show_progress=True)

# Selecting a single datapoint
selected = data.isel(time=0)

# Returns the local path where data already exists
local_path = storage_client.download(selected)
print(local_path)

# List files relative to the datapoint location
objects = storage_client.list_objects(selected)
print(objects)
```

### Datapoint fields used by this client

* `location` (required): Path to the product directory or file, relative to the configured `root`.
* `thumbnail`, `overview`, or `quicklook` (optional): Relative path used by `download_quicklook` and `quicklook`.

If quicklook metadata is present, you can access it the same way as with other storage clients:

```python Python theme={"system"}
quicklook_path = storage_client.download_quicklook(selected)
storage_client.quicklook(selected)
```

## Copernicus Data Space

The [Copernicus Data Space](https://dataspace.copernicus.eu/) is an open ecosystem that provides free instant access to data and services from the Copernicus Sentinel missions. Check out the [ASF Open Data datasets](/datasets/open-data#copernicus-data-space) that are available in Tilebox.

### Access Copernicus data

To download data products from the Copernicus Data Space after querying them via the Tilebox API, you need to [create an account](https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/auth?client_id=cdse-public\&response_type=code\&scope=openid\&redirect_uri=https%3A//dataspace.copernicus.eu/account/confirmed/1) and then generate [S3 credentials here](https://eodata-s3keysmanager.dataspace.copernicus.eu/panel/s3-credentials).

The following code snippet demonstrates how to query and download Copernicus data using the Tilebox Python SDK.

```python Python highlight={4,8-12,25} theme={"system"}
from pathlib import Path

from tilebox.datasets import Client
from tilebox.storage import CopernicusStorageClient

# Creating clients
client = Client()
storage_client = CopernicusStorageClient(
    access_key="YOUR_ACCESS_KEY",
    secret_access_key="YOUR_SECRET_ACCESS_KEY",
    cache_directory=Path("./data")
)

# Choosing the dataset and collection
s2_dataset = client.dataset("open_data.copernicus.sentinel2_msi")
collection = s2_dataset.collection("S2A_S2MSI2A")

# Loading metadata
s2_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)

# Selecting a data point to download
selected = s2_data.isel(time=0)  # index 0 selected

# Downloading the data
downloaded_data = storage_client.download(selected)

print(f"Downloaded granule: {downloaded_data.name} to {downloaded_data}")
print("Contents: ")
for content in downloaded_data.iterdir():
    print(f" - {content.relative_to(downloaded_data)}")
```

```plaintext Output theme={"system"}
Downloaded granule: S2A_MSIL2A_20240801T002611_N0511_R102_T58WET_20240819T170544.SAFE to data/Sentinel-2/MSI/L2A/2024/08/01/S2A_MSIL2A_20240801T002611_N0511_R102_T58WET_20240819T170544.SAFE
Contents:
  - manifest.safe
  - GRANULE
  - INSPIRE.xml
  - MTD_MSIL2A.xml
  - DATASTRIP
  - HTML
  - rep_info
  - S2A_MSIL2A_20240801T002611_N0511_R102_T58WET_20240819T170544-ql.jpg
```

### Partial product downloads

For cases where only a subset of the available file objects for a product is needed, you may restrict your download to just that subset. First, list available objects using `list_objects`, filter them, and then download using `download_objects`.

For example, a Sentinel-2 L2A product includes many files such as metadata, different bands in multiple resolutions, masks, and quicklook images. The following example shows how to download only specific files from a Sentinel-2 L2A product.

```python Python highlight={6, 17} theme={"system"}
s2_dataset = client.dataset("open_data.copernicus.sentinel2_msi")
collection = s2_dataset.collection("S2A_S2MSI2A")
s2_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)
selected = s2_data.isel(time=0)  # download the first granule in the given time range

objects = storage_client.list_objects(selected)
print(f"Granule {selected.granule_name.item()} consists of {len(objects)} individual objects.")

# only select specific objects to download
want_products = ["B02_10m", "B03_10m", "B08_10m"]
objects = [obj for obj in objects if any(prod in obj for prod in want_products)]  # remove all other objects
print(f"Downloading {len(objects)} objects.")
for obj in objects:
    print(f" - {obj}")

# Finally, download the selected data
downloaded_data = storage_client.download_objects(selected, objects)
```

```plaintext Output theme={"system"}
Granule S2A_MSIL2A_20240801T002611_N0511_R102_T58WET_20240819T170544.SAFE consists of 95 individual objects.
Downloading 3 objects.
 - GRANULE/L2A_T58WET_A047575_20240801T002608/IMG_DATA/R10m/T58WET_20240801T002611_B02_10m.jp2
 - GRANULE/L2A_T58WET_A047575_20240801T002608/IMG_DATA/R10m/T58WET_20240801T002611_B03_10m.jp2
 - GRANULE/L2A_T58WET_A047575_20240801T002608/IMG_DATA/R10m/T58WET_20240801T002611_B08_10m.jp2
```

### Quicklook images

Many Copernicus products include a quicklook image. The Tilebox storage client includes support for displaying these quicklook images directly when running in an interactive environment such as a Jupyter notebook.

```python Python highlight={6} theme={"system"}
s2_dataset = client.dataset("open_data.copernicus.sentinel2_msi")
collection = s2_dataset.collection("S2A_S2MSI2A")
s2_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)
selected = s2_data.isel(time=0)  # download the first granule in the given time range

storage_client.quicklook(selected)
```

<Frame>
  <img alt="Quicklook image" />
</Frame>

## USGS Landsat

The [United States Geological Survey (USGS)](https://www.usgs.gov/) provides a wide range of Earth observation data, including Landsat data, which are also available as open data through Tilebox.

### Accessing Landsat data

Landsat data is available in a S3 bucket. The following code snippet demonstrates how to query and download Landsat data using the Tilebox Python SDK.

<Info>
  The USGS Landsat S3 bucket is a [requester-pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) bucket.
  This means that you will need to have an AWS account to access the data, and then have your AWS credentials configured in your environment.
  Check out the [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for more information on how to configure your credentials.
</Info>

```python Python theme={"system"}
from pathlib import Path

from tilebox.datasets import Client
from tilebox.storage import USGSLandsatStorageClient

# Creating clients
client = Client()
storage_client = USGSLandsatStorageClient()

# Choosing the dataset and collection
l9_dataset = client.dataset("open_data.usgs.landsat9_oli_tirs")
collection = l9_dataset.collection("L2_SR")

# Loading metadata
l9_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)

# Selecting a data point to download
selected = l9_data.isel(time=0)  # index 0 selected

# Downloading the data
downloaded_data = storage_client.download(selected)

print(f"Downloaded granule: {downloaded_data.name} to {downloaded_data}")
print("Contents: ")
for content in downloaded_data.iterdir():
    print(f" - {content.relative_to(downloaded_data)}")
```

```plaintext Output theme={"system"}
Downloaded granule: LC09_L2SP_088241_20240801_20240802_02_T1 to /Users/lukasbindreiter/.cache/tilebox/collection02/level-2/standard/oli-tirs/2024/088/241/LC09_L2SP_088241_20240801_20240802_02_T1
Contents: 
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_EMSD.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_QA_AEROSOL.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B7.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_QA.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B6.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B4.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B5.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B1.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B2.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B3.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_URAD.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ANG.txt
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_CDIST.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_thumb_large.jpeg
 - LC09_L2SP_088241_20240801_20240802_02_T1_thumb_small.jpeg
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_B10.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_MTL.txt
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_TRAD.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_MTL.json
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_DRAD.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_ATRAN.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_QA_RADSAT.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_stac.json
 - LC09_L2SP_088241_20240801_20240802_02_T1_MTL.xml
 - LC09_L2SP_088241_20240801_20240802_02_T1_QA_PIXEL.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_EMIS.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_ST_stac.json

```

### Partial product downloads

For cases where only a subset of the available file objects for a product is needed, you may restrict your download to just that subset. First, list available objects using `list_objects`, filter them, and then download using `download_objects`.

For example, a Landsat 9 L2 SR product includes many files such as metadata, different bands in multiple resolutions, masks, and quicklook images. The following example shows how to download only specific files from a Landsat 9 L2 SR product.

```python highlight={17} theme={"system"}
l9_dataset = client.dataset("open_data.usgs.landsat9_oli_tirs")
collection = l9_dataset.collection("L2_SR")
l9_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)
selected = l9_data.isel(time=0)

objects = storage_client.list_objects(selected)
print(f"Granule {selected.granule_name.item()} consists of {len(objects)} individual objects.")

rgb_bands = ["B4", "B3", "B2"]

objects = [obj for obj in objects if any(obj.endswith(band + ".TIF") for band in rgb_bands)]
print(f"Downloading {len(objects)} objects.")
for obj in objects:
    print(f" - {obj}")

# Finally, download the selected data
downloaded_data = storage_client.download_objects(selected, objects)
```

```plaintext Output theme={"system"}
Granule LC09_L2SP_088241_20240801_20240802_02_T1_SR consists of 27 individual objects.
Downloading 3 objects.
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B2.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B3.TIF
 - LC09_L2SP_088241_20240801_20240802_02_T1_SR_B4.TIF
```

### Quicklook images

Many USGS products include a quicklook image. The Tilebox storage client includes support for displaying these quicklook images directly when running in an interactive environment such as a Jupyter notebook.

```python Python highlight={10} theme={"system"}
# Loading metadata
l9_dataset = client.dataset("open_data.usgs.landsat9_oli_tirs")
collection = l9_dataset.collection("L2_SR")
l9_data = collection.query(temporal_extent=("2024-08-01", "2024-08-02"), show_progress=True)

# Selecting a data point to download
selected = l9_data.isel(time=0)  # index 0 selected

# Displaying a quicklook image
storage_client.quicklook(selected)
```

<Frame>
  <img alt="USGS Quicklook image" />
</Frame>

## Alaska Satellite Facility (ASF)

The [Alaska Satellite Facility (ASF)](https://asf.alaska.edu/) is a NASA-funded research center at the University of Alaska Fairbanks. Check out the [ASF Open Data datasets](/datasets/open-data#alaska-satellite-facility-asf) that are available in Tilebox.

### Accessing ASF Data

You can query ASF metadata without needing an account, as Tilebox has indexed and ingested the relevant metadata. To access and download the actual satellite products, you will need an ASF account.

You can create an ASF account in the [ASF Vertex Search Tool](https://search.asf.alaska.edu/).

The following code snippet demonstrates how to query and download ASF data using the Tilebox Python SDK.

```python Python highlight={4,9-13,27} theme={"system"}
from pathlib import Path

from tilebox.datasets import Client
from tilebox.storage import ASFStorageClient

# Creating clients
client = Client()
storage_client = ASFStorageClient(
    user="YOUR_ASF_USER",
    password="YOUR_ASF_PASSWORD",
    cache_directory=Path("./data")
)

# Choosing the dataset and collection
ers_dataset = client.dataset("open_data.asf.ers_sar")
collection = ers_dataset.collection("ERS-2")

# Loading metadata
ers_data = collection.query(temporal_extent=("2009-01-01", "2009-01-02"), show_progress=True)

# Selecting a data point to download
selected = ers_data.isel(time=0)  # index 0 selected

# Downloading the data
downloaded_data = storage_client.download(selected, extract=True)

print(f"Downloaded granule: {downloaded_data.name} to {downloaded_data}")
print("Contents: ")
for content in downloaded_data.iterdir():
    print(f" - {content.relative_to(downloaded_data)}")
```

```plaintext Output theme={"system"}
Downloaded granule: E2_71629_STD_L0_F183 to data/ASF/E2_71629_STD_F183/E2_71629_STD_L0_F183
Contents:
  - E2_71629_STD_L0_F183.000.vol
  - E2_71629_STD_L0_F183.000.meta
  - E2_71629_STD_L0_F183.000.raw
  - E2_71629_STD_L0_F183.000.pi
  - E2_71629_STD_L0_F183.000.nul
  - E2_71629_STD_L0_F183.000.ldr
```

### Quicklook images

Many ASF products include a quicklook image. The Tilebox storage client includes support for displaying these quicklook images directly when running in an interactive environment such as a Jupyter notebook.

```python Python highlight={10} theme={"system"}
# Loading metadata
ers_dataset = client.dataset("open_data.asf.ers_sar")
collection = ers_dataset.collection("ERS-2")
ers_data = collection.query(temporal_extent=("2009-01-01", "2009-01-02"), show_progress=True)

# Selecting a data point to download
selected = ers_data.isel(time=0)  # index 0 selected

# Displaying a quicklook image
storage_client.quicklook(selected)
```

<Frame>
  <img alt="ASF ERS Quicklook image" />
</Frame>

### Further Reading

<Columns>
  <Card title="Getting Started with ASF" icon="arrow-up-right-from-square" href="https://asf.alaska.edu/getstarted/" />

  <Card title="ASF Data Formats and Files" icon="arrow-up-right-from-square" href="https://asf.alaska.edu/information/data-formats/data-formats-in-depth/" />
</Columns>

## Umbra Space

[Umbra](https://umbra.space/) satellites provide high resolution Synthetic Aperture Radar (SAR) imagery from space. Check out the [Umbra datasets](/datasets/open-data#umbra-space) that are available in Tilebox.

### Accessing Umbra data

No account is needed to access Umbra data. All data is under a Creative Commons License (CC BY 4.0), allowing you to use it freely.

The following code snippet demonstrates how to query and download Umbra data using the Tilebox Python SDK.

```python Python highlight={4,9,23} theme={"system"}
from pathlib import Path

from tilebox.datasets import Client
from tilebox.storage import UmbraStorageClient

# Creating clients
client = Client()
datasets = client.datasets()
storage_client = UmbraStorageClient(cache_directory=Path("./data"))

# Choosing the dataset and collection
umbra_dataset = datasets.open_data.umbra.sar
collections = umbra_dataset.collections()
collection = collections["SAR"]

# Loading metadata
umbra_data = collection.query(temporal_extent=("2024-01-05", "2024-01-06"), show_progress=True)

# Selecting a data point to download
selected = umbra_data.isel(time=0)  # index 0 selected

# Downloading the data
downloaded_data = storage_client.download(selected)

print(f"Downloaded granule: {downloaded_data.name} to {downloaded_data}")
print("Contents: ")
for content in downloaded_data.iterdir():
    print(f" - {content.relative_to(downloaded_data)}")
```

```plaintext Output theme={"system"}
Downloaded granule: 2024-01-05-01-53-37_UMBRA-07 to data/Umbra/ad hoc/Yi_Sun_sin_Bridge_SK/6cf02931-ca2e-4744-b389-4844ddc701cd/2024-01-05-01-53-37_UMBRA-07
Contents:
 - 2024-01-05-01-53-37_UMBRA-07_SIDD.nitf
 - 2024-01-05-01-53-37_UMBRA-07_SICD.nitf
 - 2024-01-05-01-53-37_UMBRA-07_CSI-SIDD.nitf
 - 2024-01-05-01-53-37_UMBRA-07_METADATA.json
 - 2024-01-05-01-53-37_UMBRA-07_GEC.tif
 - 2024-01-05-01-53-37_UMBRA-07_CSI.tif
```

### Partial product downloads

For cases where only a subset of the available file objects for a given Umbra data point is necessary, you can limit your download to just that subset. First, list available objects using `list_objects`, filter the list, and then use `download_objects`.

The below example shows how to download only the metadata file for a given data point.

```python Python highlight={4, 15} theme={"system"}
collection = datasets.open_data.umbra.sar.collections()["SAR"]
umbra_data = collection.query(temporal_extent=("2024-01-05", "2024-01-06"), show_progress=True)
# Selecting a data point to download
selected = umbra_data.isel(time=0)  # index 0 selected

objects = storage_client.list_objects(selected)
print(f"Data point {selected.granule_name.item()} consists of {len(objects)} individual objects.")

# only select specific objects to download
objects = [obj for obj in objects if "METADATA" in obj]  # remove all other objects
print(f"Downloading {len(objects)} object.")
print(objects)

# Finally, download the selected data
downloaded_data = storage_client.download_objects(selected, objects)
```

```plaintext Output theme={"system"}
Data point 2024-01-05-01-53-37_UMBRA-07 consists of 6 individual objects.
Downloading 1 object.
['2024-01-05-01-53-37_UMBRA-07_METADATA.json']
```


# Spatio-temporal
Source: https://docs.tilebox.com/datasets/types/spatiotemporal

Spatio-temporal datasets associate each data point with both a timestamp and a geographic location on Earth, enabling queries that span time and space.

Each spatio-temporal dataset comes with a set of required and auto-generated fields for each data point.

## Required fields

While the specific data fields between different time series datasets can vary, there are common fields that all time series datasets share.

<ParamField type="Timestamp">
  The timestamp associated with each data point. Timestamps are always in UTC.
</ParamField>

<Tip>
  For indexing and querying, Tilebox truncates timestamps to millisecond precision. But Timeseries datasets may contain arbitrary custom `Timestamp` fields that store timestamps up to a nanosecond precision.
</Tip>

<ParamField type="Geometry">
  A location on the earth's surface associated with each data point. Supported geometry types are `Point`, `Polygon` and `MultiPolygon`.
</ParamField>

## Auto-generated fields

<ParamField type="UUID">
  A [universally unique identifier (UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier) that uniquely identifies each data point. IDs are generated so that sorting them lexicographically also sorts them by time.
</ParamField>

<Tip>
  IDs generated by Tilebox are deterministic, meaning that ingesting the exact same data values into the same collection will always result in the same ID.
</Tip>

<ParamField type="Timestamp">
  The time the data point was ingested into the Tilebox API.
</ParamField>

## Creating a spatio-temporal dataset

To create a spatio-temporal dataset, use the [Tilebox Console](/console) and select `Spatio-temporal Dataset` as the dataset type. The required and auto-generated fields
already outlined will be automatically added to the dataset schema.

## Spatio-temporal queries

Spatio-temporal datasets support efficient time-based and spatially filtered queries. To query a specific location in a given time interval,
specify a time range and a geometry when [querying data points](/datasets/query/filter-by-location) from a dataset or a collection.

## Geometries

Handling Geometries can traditionally be a bit tricky, especially when working with geometries that cross the antimeridian or cover a pole.
Tilebox is designed to take away most of the friction involved in this, but it's still recommended to follow the [best practices for handling geometries](/datasets/geometries).


# Timeseries
Source: https://docs.tilebox.com/datasets/types/timeseries

Timeseries datasets associate each data point with a specific timestamp, making them well suited for organizing and querying time-ordered observations.

Each timeseries dataset comes with a set of required and auto-generated fields for each data point.

## Required fields

While the specific data fields between different time series datasets can vary, there are common fields that all time series datasets share.

<ParamField type="Timestamp">
  The timestamp associated with each data point. Timestamps are always in UTC.
</ParamField>

<Tip>
  For indexing and querying, Tilebox truncates timestamps to millisecond precision. But Timeseries datasets may contain arbitrary custom `Timestamp` fields that store timestamps up to a nanosecond precision.
</Tip>

## Auto-generated fields

<ParamField type="UUID">
  A [universally unique identifier (UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier) that uniquely identifies each data point. IDs are generated so that sorting them lexicographically also sorts them by time.
</ParamField>

<Tip>
  IDs generated by Tilebox are deterministic, meaning that ingesting the exact same data values into the same collection will always result in the same ID.
</Tip>

<ParamField type="Timestamp">
  The time the data point was ingested into the Tilebox API.
</ParamField>

## Creating a timeseries dataset

To create a timeseries dataset, use the [Tilebox Console](/console) and select `Timeseries Dataset` as the dataset type. The required and auto-generated fields
already outlined will be automatically added to the dataset schema.

## Time-based queries

Timeseries datasets support time-based queries. To query a specific time interval, specify a time range when [querying data](/datasets/query/filter-by-time) from a dataset or a collection.


# Tilebox Cookbook
Source: https://docs.tilebox.com/guides/cookbook

Task-focused Tilebox guides for datasets, workflows, and operations.

<div>
  <div>
    <p>
      User guides
    </p>

    <h1>
      Tilebox Cookbook
    </h1>

    <p>
      Recipes for going from question to running system: query satellite catalogs, ingest your own data, ship workflows, and debug what happened.
    </p>
  </div>
</div>


# Access Copernicus data
Source: https://docs.tilebox.com/guides/datasets/access-sentinel2-data

Download Copernicus Data Space products with the Tilebox Copernicus storage client, using Sentinel-2 as an example.

Use this guide when you already have a Copernicus datapoint from a Tilebox metadata query and want to access the product files behind it. The example uses Sentinel-2 Level-2A data, but the same storage client pattern applies to Copernicus products supported by Tilebox.

Tilebox indexes product metadata as datasets. Product files remain in the Copernicus Data Space Ecosystem, so file access uses the `CopernicusStorageClient` with Copernicus S3 credentials.

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).
* You have a [Copernicus Data Space](https://dataspace.copernicus.eu/) account.
* You have generated Copernicus [S3 credentials](https://eodata-s3keysmanager.dataspace.copernicus.eu/panel/s3-credentials).

```bash theme={"system"}
uv add tilebox shapely
```

## Select a Sentinel-2 datapoint

Start with a small metadata query and select one datapoint to access. For a deeper guide to open data discovery and metadata filtering, see [Query open data metadata](/guides/datasets/query-satellite-data).

```python Python theme={"system"}
from shapely import Polygon
from tilebox.datasets import Client

area = Polygon(
    [
        (-109.05, 37.0),
        (-102.05, 37.0),
        (-102.05, 41.0),
        (-109.05, 41.0),
        (-109.05, 37.0),
    ]
)

client = Client()
collection = client.dataset("open_data.copernicus.sentinel2_msi").collection("S2A_S2MSI2A")

scenes = collection.query(
    temporal_extent=("2025-10-01", "2025-11-01"),
    spatial_extent=area,
    show_progress=True,
)

selected = scenes.where(scenes.cloud_cover < 10, drop=True).isel(time=0)
print(selected.granule_name.item())
```

## Create the Copernicus storage client

Create a `CopernicusStorageClient` with your Copernicus S3 credentials. The optional `cache_directory` controls where downloaded files are stored locally.

```python Python theme={"system"}
from pathlib import Path

from tilebox.storage import CopernicusStorageClient

storage = CopernicusStorageClient(
    access_key="YOUR_COPERNICUS_ACCESS_KEY",
    secret_access_key="YOUR_COPERNICUS_SECRET_ACCESS_KEY",
    cache_directory=Path("./data"),
)
```

<Note>
  These credentials are Copernicus Data Space S3 credentials, not your Tilebox API key.
</Note>

## Download the complete product

Use `download` when you need the complete Sentinel-2 product directory. The storage client resolves the product location from the Tilebox datapoint metadata and downloads the matching files into the local cache directory.

```python Python theme={"system"}
product_path = storage.download(selected)

print(f"Downloaded {product_path.name} to {product_path}")
print("Contents:")
for path in product_path.iterdir():
    print(f"- {path.relative_to(product_path)}")
```

```plaintext Output theme={"system"}
Downloaded S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842.SAFE to data/Sentinel-2/MSI/L2A/2025/10/02/S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842.SAFE
Contents:
- manifest.safe
- GRANULE
- INSPIRE.xml
- MTD_MSIL2A.xml
- DATASTRIP
- HTML
- rep_info
- S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842-ql.jpg
```

## Download selected product files

Sentinel-2 products contain many files, including metadata, masks, quicklook images, and bands at different resolutions. Use `list_objects` and `download_objects` when you only need specific files.

```python Python theme={"system"}
objects = storage.list_objects(selected)

wanted_bands = ["B02_10m", "B03_10m", "B04_10m", "B08_10m"]
band_objects = [
    obj for obj in objects
    if any(band in obj for band in wanted_bands)
]

for obj in band_objects:
    print(obj)

downloaded_files = storage.download_objects(selected, band_objects)
print(downloaded_files)
```

Use this pattern when a workflow only needs a few bands or metadata files. It reduces transfer time and local storage compared with downloading the full `.SAFE` product.

## Preview the product

Many Copernicus products include a quicklook image. In a notebook, use `quicklook` to display the product preview without downloading the full product first.

```python Python theme={"system"}
storage.quicklook(selected)
```

<Frame>
  <img alt="Sentinel-2 quicklook image" />
</Frame>

## Next steps

<Columns>
  <Card title="Query open data metadata" icon="satellite" href="/guides/datasets/query-satellite-data">
    Find Copernicus products by time, location, and metadata fields.
  </Card>

  <Card title="Storage clients" icon="hard-drive" href="/datasets/storage/clients">
    Learn about the other Tilebox storage clients for open data products.
  </Card>

  <Card title="Access USGS Landsat data" icon="satellite-dish" href="/guides/datasets/access-usgs-landsat-data">
    Download Landsat product files with the USGS Landsat storage client.
  </Card>
</Columns>


# Access USGS Landsat data
Source: https://docs.tilebox.com/guides/datasets/access-usgs-landsat-data

Download USGS Landsat products with the Tilebox Landsat storage client, using Landsat 8 as an example.

Use this guide when you already have a Landsat datapoint from a Tilebox metadata query and want to access the product files behind it. The example uses Landsat 8 Collection 2 Level-2 surface reflectance data.

Tilebox indexes Landsat metadata as datasets. Product files remain in the USGS public cloud archive, so file access uses the `USGSLandsatStorageClient` and your AWS requester-pays setup.

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).
* You have AWS credentials configured in your environment.
* Your AWS account can access [requester-pays S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html).

```bash theme={"system"}
uv add tilebox shapely
```

<Info>
  USGS Landsat data is stored in a requester-pays S3 bucket. AWS charges for requests and data transfer according to your AWS account settings.
</Info>

## Select a Landsat 8 datapoint

Start with a small metadata query and select one datapoint to access. For a deeper guide to open data discovery and metadata filtering, see [Query open data metadata](/guides/datasets/query-satellite-data).

```python Python theme={"system"}
from shapely import Polygon
from tilebox.datasets import Client

area = Polygon(
    [
        (-109.05, 37.0),
        (-102.05, 37.0),
        (-102.05, 41.0),
        (-109.05, 41.0),
        (-109.05, 37.0),
    ]
)

client = Client()
collection = client.dataset("open_data.usgs.landsat8_oli_tirs").collection("L2_SR")

scenes = collection.query(
    temporal_extent=("2024-08-01", "2024-08-15"),
    spatial_extent=area,
    show_progress=True,
)

selected = scenes.where(scenes.cloud_cover < 10, drop=True).isel(time=0)
print(selected.granule_name.item())
```

## Create the Landsat storage client

Create a `USGSLandsatStorageClient`. The client uses AWS credentials from your environment, such as `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` when needed.

```python Python theme={"system"}
from tilebox.storage import USGSLandsatStorageClient

storage = USGSLandsatStorageClient()
```

## Download the complete product

Use `download` when you need the complete Landsat product directory. The storage client resolves the product location from the Tilebox datapoint metadata and downloads the matching files into the local cache.

```python Python theme={"system"}
product_path = storage.download(selected)

print(f"Downloaded {product_path.name} to {product_path}")
print("Contents:")
for path in product_path.iterdir():
    print(f"- {path.relative_to(product_path)}")
```

```plaintext Output theme={"system"}
Downloaded LC08_L2SP_033033_20240808_20240814_02_T1 to ~/.cache/tilebox/collection02/level-2/standard/oli-tirs/2024/033/033/LC08_L2SP_033033_20240808_20240814_02_T1
Contents:
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B1.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B2.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B3.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B4.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B5.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B6.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B7.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_QA_PIXEL.TIF
- LC08_L2SP_033033_20240808_20240814_02_T1_MTL.json
- LC08_L2SP_033033_20240808_20240814_02_T1_thumb_small.jpeg
```

## Download selected product files

Landsat products contain surface reflectance bands, quality masks, thermal bands, metadata, and preview images. Use `list_objects` and `download_objects` when you only need specific files.

```python Python theme={"system"}
objects = storage.list_objects(selected)

rgb_bands = ["B4", "B3", "B2"]
rgb_objects = [
    obj for obj in objects
    if any(obj.endswith(f"_{band}.TIF") for band in rgb_bands)
]

for obj in rgb_objects:
    print(obj)

downloaded_files = storage.download_objects(selected, rgb_objects)
print(downloaded_files)
```

Use this pattern when a workflow only needs a few bands, masks, or metadata files. It reduces transfer time and local storage compared with downloading the full product.

## Preview the product

Many Landsat products include a thumbnail image. In a notebook, use `quicklook` to display the product preview without downloading the full product first.

```python Python theme={"system"}
storage.quicklook(selected)
```

<Frame>
  <img alt="USGS Landsat quicklook image" />
</Frame>

## Next steps

<Columns>
  <Card title="Query open data metadata" icon="satellite" href="/guides/datasets/query-satellite-data">
    Find Landsat products by time, location, and metadata fields.
  </Card>

  <Card title="Storage clients" icon="hard-drive" href="/datasets/storage/clients">
    Learn about the other Tilebox storage clients for open data products.
  </Card>
</Columns>


# Build a spatio-temporal catalog
Source: https://docs.tilebox.com/guides/datasets/build-spatiotemporal-catalog

Create a custom spatio-temporal dataset catalog with the Python SDK, document its schema, ingest geospatial metadata, and query it by time and location.

Use a spatio-temporal dataset when each datapoint has both a time and a geometry. This is useful for internal imagery catalogs, derived products, ground truth data, regions of interest, and processing outputs that need geospatial lookup.

This guide creates an imagery catalog from code. You will define the dataset schema with the Python SDK, add field descriptions and examples for generated schema documentation, create a collection, ingest geospatial metadata, and query the catalog by time and location.

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).

```bash theme={"system"}
uv add tilebox geopandas shapely
```

## Define the catalog schema

Start by choosing the spatio-temporal dataset kind and the custom fields for your catalog. Tilebox adds the required `time`, `id`, `ingestion_time`, and `geometry` fields automatically.

The example catalog tracks imagery products with a provider product ID, a storage location, cloud cover, and processing level. Field descriptions and example values become part of the generated schema documentation.

```python Python theme={"system"}
from tilebox.datasets import Client
from tilebox.datasets.data.datasets import DatasetKind

client = Client()

fields = [
    {
        "name": "product_id",
        "type": str,
        "description": "Stable product or scene identifier from the source catalog.",
        "example_value": "LC08_L2SP_033033_20240808_20240814_02_T1",
    },
    {
        "name": "location",
        "type": str,
        "description": "Storage path, object key, or provider-specific product location.",
        "example_value": "s3://example-bucket/landsat/LC08_L2SP_033033_20240808_20240814_02_T1",
    },
    {
        "name": "cloud_cover",
        "type": float,
        "description": "Cloud cover percentage for the product footprint.",
        "example_value": "3.2",
    },
    {
        "name": "processing_level",
        "type": str,
        "description": "Processing level or product type assigned by the source provider.",
        "example_value": "L2_SR",
    },
]
```

Use field names that are stable and descriptive. Changing or removing fields after ingesting datapoints requires emptying the affected collections first, because existing datapoints must continue to match the dataset schema.

## Create the dataset

Call `create_or_update_dataset` with the dataset kind, code name, field list, and display name. The code name becomes the stable identifier used in SDK calls.

```python Python theme={"system"}
dataset = client.create_or_update_dataset(
    kind=DatasetKind.SPATIOTEMPORAL,
    code_name="internal_imagery_catalog",
    fields=fields,
    name="Internal imagery catalog",
)

print(dataset)
```

If a dataset with the same code name already exists, `create_or_update_dataset` updates it instead of creating a duplicate. This makes the snippet safe to keep in a setup script.

<Note>
  The Python SDK currently sets the dataset display name and schema. Field-level `description` and `example_value` entries populate the generated schema documentation. Use the Tilebox Console when you want to add rich Markdown documentation to the dataset page.
</Note>

## Inspect the generated schema documentation

Tilebox uses the dataset kind and field annotations to document the schema. Required fields are added by the dataset kind, and your custom fields appear with their descriptions and examples.

For this catalog, the complete schema includes:

| Field              | Type     | Purpose                                    |
| ------------------ | -------- | ------------------------------------------ |
| `time`             | Required | Timestamp associated with the datapoint.   |
| `id`               | Required | Tilebox-generated UUID for the datapoint.  |
| `ingestion_time`   | Required | Time when Tilebox ingested the datapoint.  |
| `geometry`         | Required | Geometry used for spatial queries.         |
| `product_id`       | Custom   | Stable product or scene identifier.        |
| `location`         | Custom   | Storage path or provider product location. |
| `cloud_cover`      | Custom   | Cloud cover percentage for filtering.      |
| `processing_level` | Custom   | Provider processing level or product type. |

The descriptions and example values you provided in the SDK call appear in the dataset schema documentation.

## Add richer dataset documentation

Use field descriptions for schema-level documentation. Use the Console documentation editor when you want longer Markdown documentation for the dataset, such as provenance notes, quality caveats, ingestion rules, or examples for downstream users.

<Frame>
  <img alt="Tilebox Console dataset documentation editor" />

  <img alt="Tilebox Console dataset documentation editor" />
</Frame>

Open the dataset in the Console, click the edit pencil on the documentation section, and add Markdown content. A short documentation block often includes:

```md Markdown theme={"system"}
# Internal imagery catalog

This dataset indexes analysis-ready imagery products used by the operations team.

## Source

Products are copied from the provider archive after validation.

## Usage notes

Use `cloud_cover < 10` for workflows that require mostly cloud-free scenes.
```

## Create a collection

After creating the dataset, create a collection to hold datapoints. Collections let you organize datapoints within the same schema, for example by provider, product family, or processing pipeline.

```python Python theme={"system"}
collection = dataset.get_or_create_collection("landsat_level_2")
print(collection)
```

## Prepare datapoints

Load your source metadata into a GeoDataFrame. The geometry column should contain the footprint for each datapoint.

```python Python theme={"system"}
import geopandas as gpd

products = gpd.read_parquet("products.geoparquet")
products = products.rename(
    columns={
        "timestamp": "time",
        "scene": "product_id",
        "path": "location",
    }
)

products = products[
    ["time", "geometry", "product_id", "location", "cloud_cover", "processing_level"]
]
```

## Ingest the catalog

Ingest the prepared records into a collection.

```python Python theme={"system"}
collection.ingest(products)
```

## Query by time and location

After ingestion, use the same query model as Tilebox open data catalogs.

```python Python theme={"system"}
from shapely import box

area = box(11.0, 46.0, 12.0, 47.0)

matches = collection.query(
    temporal_extent=("2026-01-01", "2026-02-01"),
    spatial_extent=area,
)
```

## Next steps

<Columns>
  <Card title="Spatio-temporal datasets" icon="globe" href="/datasets/types/spatiotemporal">
    Learn the required fields and query behavior.
  </Card>

  <Card title="Ingest from common file formats" icon="file-binary" href="/guides/datasets/ingest-format">
    Load CSV, Parquet, GeoParquet, and NetCDF data before ingestion.
  </Card>

  <Card title="Ingest into a spatio-temporal catalog" icon="up-from-bracket" href="/guides/datasets/ingest-into-spatiotemporal-catalog">
    Prepare GeoParquet metadata and ingest it into this catalog.
  </Card>
</Columns>


# Ingesting from common file formats
Source: https://docs.tilebox.com/guides/datasets/ingest-format

Convert data from common geospatial and tabular file formats into Tilebox datasets by loading them as DataFrames or xarray Datasets and ingesting directly.

For ingesting data from common file formats, it's recommend to use the [Tilebox Python SDK](/sdks/python/install), since
it provides out-of-the-box support for reading many common formats through third party libraries for loading data as either
`pandas.DataFrame` or `xarray.Dataset`, which can then be directly ingested into Tilebox.

## Reading and previewing the data

To ingest data from a file, you first need to read it into a `pandas.DataFrame` or an `xarray.Dataset`.
How that can be achieved depends on the file format. The following sections show examples for a couple of common
file formats.

### CSV

Comma-separated values (CSV) is a common file format for tabular data. It's widely used in data science. Tilebox
supports CSV ingestion using the `pandas.read_csv` function.

Assuming you have a CSV file named `data.csv` with the following content. If you want to follow along, you can
download the file [here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.csv).

```csv ingestion_data.csv theme={"system"}
time,value,sensor,precise_time,sensor_history,some_unwanted_column
2025-03-28T11:44:23Z,45.16,A,2025-03-28T11:44:23.345761444Z,"[-12.15, 13.45, -8.2, 16.5, 45.16]","Unsupported"
2025-03-28T11:45:19Z,273.15,B,2025-03-28T11:45:19.128742312Z,"[300.16, 280.12, 273.15]","Unsupported"
```

<CodeGroup>
  ```python Python theme={"system"}
  import pandas as pd

  data = pd.read_csv("ingestion_data.csv")
  ```
</CodeGroup>

### Parquet

[Apache Parquet](https://parquet.apache.org/) is an open source, column-oriented data file format designed for efficient data storage and retrieval.
Tilebox supports Parquet ingestion using the `pandas.read_parquet` function.

The parquet file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.parquet).

<CodeGroup>
  ```python Python theme={"system"}
  import pandas as pd

  data = pd.read_parquet("ingestion_data.parquet")
  ```
</CodeGroup>

### GeoParquet

[GeoParquet](https://geoparquet.org/) is an extension of the Parquet file format, adding geospatial
features support to Parquet. Tilebox supports GeoParquet ingestion using the `geopandas.read_parquet` function.

The GeoParquet file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/modis_MCD12Q1.geoparquet).

<CodeGroup>
  ```python Python theme={"system"}
  import geopandas as gpd

  data = gpd.read_parquet("modis_MCD12Q1.geoparquet")
  ```
</CodeGroup>

<Tip>
  For a step-by-step guide to ingesting a GeoParquet file, see [Ingest into a spatio-temporal catalog](/guides/datasets/ingest-into-spatiotemporal-catalog).
</Tip>

### Feather

[Feather](https://arrow.apache.org/docs/python/feather.html) is a file format originating from the Apache Arrow project,
designed for storing tabular data in a fast and memory-efficient way. Tilebox supports Feather ingestion using the `pandas.read_feather` function.

The feather file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.feather).

<CodeGroup>
  ```python Python theme={"system"}
  import pandas as pd

  data = pd.read_feather("ingestion_data.feather")
  ```
</CodeGroup>

## Mapping columns to dataset fields

Once data is read into a `pandas.DataFrame` or an `xarray.Dataset`, it can be ingested into Tilebox directly.
The column names of the `pandas.DataFrame` or the variables and coordinates of the `xarray.Dataset` are mapped to the fields of the
Tilebox dataset to ingest into.

Depending on how closely the column names or variable/coordinate names match the field names in the Tilebox dataset,
you might need to rename some columns/variables/coordinates before ingestion.

### Renaming fields

<CodeGroup>
  ```python Pandas theme={"system"}
  data = data.rename({"precise_time": "measurement_time"})
  ```

  ```python Xarray theme={"system"}
  data = data.rename({"precise_time": "measurement_time"})
  ```
</CodeGroup>

## Dropping fields

In case you want to skip certain columns/variables/coordinates entirely, you can drop them before ingestion.

<CodeGroup>
  ```python Pandas theme={"system"}
  data = data.drop(columns=["some_unwanted_column"])
  ```

  ```python Xarray theme={"system"}
  data = data.drop_vars(["some_unwanted_variable"])
  ```
</CodeGroup>

## Ingesting the data

Once the data is in the correct format, you can ingest it into Tilebox.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  # my_custom_dataset has a schema that matches the data we want to ingest
  dataset = client.dataset("my_org.my_custom_dataset")

  collection = dataset.get_or_create_collection("Measurements")
  collection.ingest(data)
  ```
</CodeGroup>


# Ingest into a spatio-temporal catalog
Source: https://docs.tilebox.com/guides/datasets/ingest-into-spatiotemporal-catalog

Prepare GeoParquet metadata and ingest it into an existing Tilebox spatio-temporal catalog.

Use this guide after [Build a spatio-temporal catalog](/guides/datasets/build-spatiotemporal-catalog). It assumes you already created a spatio-temporal dataset and now want to load geospatial metadata into one of its collections.

The example starts from a GeoParquet file, reshapes it to match the catalog schema, ingests it into Tilebox, and runs a time and location query against the new collection.

<Note>
  If your source data uses a different file format, see [Ingesting from common file formats](/guides/datasets/ingest-format) for examples of loading CSV, Parquet, GeoParquet, and NetCDF data before ingestion.
</Note>

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).
* You have created the catalog from [Build a spatio-temporal catalog](/guides/datasets/build-spatiotemporal-catalog), or an equivalent spatio-temporal dataset with matching fields.

```bash theme={"system"}
uv add tilebox geopandas lonboard shapely
```

## Download the example metadata

The example metadata is available as a [GeoParquet](https://geoparquet.org/) file:

```bash theme={"system"}
curl -L \
  -o modis_MCD12Q1.geoparquet \
  https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/modis_MCD12Q1.geoparquet
```

This file contains MODIS land cover product metadata, including timestamps and product footprints.

## Read and preview the source data

Read the GeoParquet file with Geopandas. The resulting `GeoDataFrame` includes a `geometry` column, which Tilebox uses for spatial indexing in spatio-temporal datasets.

```python Python theme={"system"}
import geopandas as gpd

source = gpd.read_parquet("modis_MCD12Q1.geoparquet")
source.head(5)
```

```plaintext Output theme={"system"}
                       time                  end_time                    granule_name                                           geometry  horizontal_tile_number  vertical_tile_number   tile_id
0 2001-01-01 00:00:00+00:00 2001-12-31 23:59:59+00:00  MCD12Q1.A2001001.h00v08...  POLYGON ((-180 10, -180 0, -170 0, ...                       0                     8  51000008
1 2001-01-01 00:00:00+00:00 2001-12-31 23:59:59+00:00  MCD12Q1.A2001001.h00v09...  POLYGON ((-180 0, -180 -10, ...                            0                     9  51000009
```

You can inspect the footprints before ingestion with `lonboard`.

```python Python theme={"system"}
from lonboard import viz

viz(source, map_kwargs={"show_tooltip": True})
```

<Frame>
  <img alt="Explore the MODIS dataset" />

  <img alt="Explore the MODIS dataset" />
</Frame>

## Match the catalog schema

Prepare a DataFrame with the fields required by the catalog. This example targets the schema from [Build a spatio-temporal catalog](/guides/datasets/build-spatiotemporal-catalog): `time`, `geometry`, `product_id`, `location`, `cloud_cover`, and `processing_level`.

```python Python theme={"system"}
products = source.copy()

products["product_id"] = products["granule_name"]
products["location"] = products["granule_name"].map(
    lambda name: f"modis://MCD12Q1/{name}"
)
products["cloud_cover"] = 0.0
products["processing_level"] = "MCD12Q1"

products = products[
    ["time", "geometry", "product_id", "location", "cloud_cover", "processing_level"]
]

products.head(5)
```

Keep the DataFrame columns aligned with the dataset schema. Required fields such as `id` and `ingestion_time` are generated by Tilebox during ingestion, so you do not include them in the input DataFrame.

## Connect to the catalog collection

Access the catalog dataset and create or reuse a collection for the MODIS products.

```python Python theme={"system"}
from tilebox.datasets import Client

client = Client()
dataset = client.dataset("internal_imagery_catalog")
collection = dataset.get_or_create_collection("modis_land_cover")
```

Replace `internal_imagery_catalog` with the code name of your catalog if you used a different value in the previous guide.

## Ingest the products

Ingest the prepared DataFrame into the collection. Tilebox validates each row against the dataset schema before storing it.

```python Python theme={"system"}
datapoint_ids = collection.ingest(products)
print(f"Successfully ingested {len(datapoint_ids)} datapoints.")
```

```plaintext Output theme={"system"}
Successfully ingested 7245 datapoints.
```

## Query the ingested catalog

After ingestion, query the collection by time and location. The query model is the same one used by Tilebox open data catalogs.

```python Python theme={"system"}
from shapely import Polygon

area = Polygon(
    [
        (-124.45, 49.19),
        (-120.88, 29.31),
        (-66.87, 24.77),
        (-65.34, 47.84),
        (-124.45, 49.19),
    ]
)

matches = collection.query(
    temporal_extent=("2015-01-01", "2020-01-01"),
    spatial_extent=area,
)

matches[["product_id", "processing_level", "location"]]
```

```plaintext Output theme={"system"}
<xarray.Dataset> Size: 18kB
Dimensions:           (time: 110)
Coordinates:
  * time              (time) datetime64[ns] 2015-01-01 ... 2019-01-01
Data variables:
    product_id        (time) object 'MCD12Q1.A2015001.h10v03...' ...
    processing_level  (time) object 'MCD12Q1' 'MCD12Q1' ...
    location          (time) object 'modis://MCD12Q1/MCD12Q1.A2015001...' ...
```

## View the data in the Console

You can also inspect ingested datapoints in the Tilebox Console. Open the dataset, select the collection, and click a datapoint to inspect its fields and geometry.

<Frame>
  <img alt="Explore the MODIS dataset" />

  <img alt="Explore the MODIS dataset" />
</Frame>

## Next steps

<Columns>
  <Card title="Build a spatio-temporal catalog" icon="globe" href="/guides/datasets/build-spatiotemporal-catalog">
    Create and document the catalog schema used by this guide.
  </Card>

  <Card title="Query data" icon="server" href="/datasets/query/querying-data">
    Learn more about querying datasets by time, location, collection, and ID.
  </Card>

  <Card title="Ingest from common file formats" icon="file-binary" href="/guides/datasets/ingest-format">
    Load CSV, Parquet, GeoParquet, and NetCDF data before ingestion.
  </Card>
</Columns>


# Query open satellite data
Source: https://docs.tilebox.com/guides/datasets/query-satellite-data

Explore available Tilebox open data catalogs and query Sentinel-2 metadata by time and location.

Use this guide when you want to find satellite products in Tilebox open data catalogs before downloading any files. You will first inspect the available open data datasets, then query Sentinel-2 metadata by time and location.

Tilebox Datasets stores searchable metadata for open Earth observation catalogs. Metadata queries are the fastest way to narrow a large catalog to the scenes that match your workflow, notebook, or agent task.

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).

```bash theme={"system"}
uv add tilebox shapely
```

## Explore available open data datasets

Tilebox exposes open data catalogs through the same dataset API as your private datasets. To get a list of available open data satellite datasets, run the following snippet.

```python Python theme={"system"}
from tilebox.datasets import Client

client = Client()
datasets = client.datasets()
print(datasets.open_data)
```

The output groups datasets by provider. Open data datasets include Copernicus Sentinel missions, USGS Landsat products, ASF SAR products, and other public catalogs that Tilebox has indexed.

```plaintext Output theme={"system"}
asf:
    ers_sar: European Remote Sensing Satellite (ERS) Synthetic Aperture Radar ...
copernicus:
    sentinel1_sar: The Sentinel-1 mission is the European Radar Observatory ...
    sentinel2_msi: Sentinel-2 is equipped with an optical instrument payload ...
    sentinel3_olci: OLCI (Ocean and Land Colour Instrument) is an optical ...
    ...
usgs:
    ...
    landsat8_oli_tirs: Landsat-8 Operational Land Imager and Thermal Infrared ...
    landsat9_oli_tirs: Landsat-9 Operational Land Imager and Thermal Infrared ...
```

You can also browse open data datasets in the [Tilebox Console](https://console.tilebox.com/datasets/open-data) when you want descriptions, provider details, the dataset schema, and available collections before writing code.

## Select the Sentinel-2 catalog

Access the Sentinel-2 MSI dataset by its slug. The dataset contains collections for Sentinel-2 products such as `S2A_S2MSI2A`.

```python Python theme={"system"}
sentinel2 = client.dataset("open_data.copernicus.sentinel2_msi")

for name, collection in sentinel2.collections().items():
    print(name, collection)
```

## Define the search area

Create a polygon for the area you want to inspect. This example uses a bounding box around Colorado.

```python Python theme={"system"}
from shapely import Polygon

area = Polygon(
    [
        # lon, lat
        (-109.05, 37.0),
        (-102.05, 37.0),
        (-102.05, 41.0),
        (-109.05, 41.0),
        # close the square (repeat the first element)
        (-109.05, 37.0),
    ]
)
```

## Query Sentinel-2 metadata

Query the Sentinel-2 Level-2A collection by time and location. This returns metadata for matching scenes; it does not download image products.

```python Python theme={"system"}
collection = sentinel2.collection("S2A_S2MSI2A")

scenes = collection.query(
    temporal_extent=("2025-10-01", "2025-11-01"),
    spatial_extent=area,
    show_progress=True,
)

print(scenes[["granule_name", "processing_level", "product_type"]])
```

The result is an `xarray.Dataset` containing scene metadata. Use it to inspect candidate scenes, filter by metadata fields, or pass selected datapoints to a workflow task.

## Filter the metadata result

Metadata results behave like regular `xarray.Dataset` objects. You can filter, sort, or select scenes before deciding what to process next.

```python Python theme={"system"}
low_cloud = scenes.where(scenes.cloud_cover < 10, drop=True)
latest = low_cloud.sortby("time").isel(time=-1)

print(latest.granule_name.item())
print(latest.cloud_cover.item())
```

Metadata queries do not download product files. Use a [storage client](/datasets/storage/clients) when you want to read or download the files referenced by a datapoint.

## Next steps

<Columns>
  <Card title="Access Copernicus data" icon="database" href="/guides/datasets/access-sentinel2-data">
    Download Copernicus product files with the storage client.
  </Card>

  <Card title="Storage clients" icon="hard-drive" href="/datasets/storage/clients">
    Configure provider-specific clients for product access.
  </Card>
</Columns>


# Configure cron automations
Source: https://docs.tilebox.com/guides/operations/configure-cron-automations

Register a workflow automation that submits jobs on a cron schedule.

Use cron automations when a workflow should run on a schedule, such as hourly catalog checks, daily quality-control jobs, or recurring data processing.

Tilebox automations submit workflow jobs when their trigger condition matches. A runner on the target cluster still needs to execute the submitted tasks.

## Define the automation task

Create a task that can run from a scheduled trigger.

```python Python theme={"system"}
from tilebox.workflows import ExecutionContext
from tilebox.workflows.automations import CronTask


class DailyCatalogCheck(CronTask):
    def execute(self, context: ExecutionContext) -> None:
        context.logger.info("Running daily catalog check", trigger=str(self.trigger))
```

## Register the cron automation

Use the automation client to register a schedule and target cluster.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()

automations.create_cron_automation(
    "daily-catalog-check",
    DailyCatalogCheck(),
    cron_schedules=["0 6 * * *"],
)
```

## Keep a runner available

The automation submits jobs. It does not execute them by itself. Start a runner that can execute the task identifier.

```python Python theme={"system"}
runner = client.runner(tasks=[DailyCatalogCheck])
runner.run_forever()
```

## Next steps

<Columns>
  <Card title="Cron triggers" icon="clock" href="/workflows/automations/cron">
    Read the full cron automation reference.
  </Card>

  <Card title="Monitor workflow runs" icon="eye" href="/guides/operations/monitor-workflow-runs">
    Inspect jobs created by automations.
  </Card>
</Columns>


# Configure storage-event automations
Source: https://docs.tilebox.com/guides/operations/configure-storage-event-automations

Register a workflow automation that submits jobs when files are created or modified in a storage location.

Use storage-event automations when new or changed files should trigger workflow work, such as processing new uploads in a bucket or reacting to generated products.

Tilebox submits a job when a matching storage event occurs. A runner on the target cluster still needs to execute the submitted task.

## Define the storage-event task

Create a task that handles the event payload.

```python Python theme={"system"}
from tilebox.workflows import ExecutionContext
from tilebox.workflows.automations import StorageEventTask, StorageEventType


class ProcessUploadedObject(StorageEventTask):
    head_bytes: int = 64

    def execute(self, context: ExecutionContext) -> None:
        if self.trigger.type == StorageEventType.CREATED:
            path = self.trigger.location
            context.logger.info(
                "Processing uploaded object",
                path=path,
            )
            data = self.trigger.storage.read(path)
            context.logger.info("Read object data", size_bytes=len(data))
```

## Register the storage location

Register or select the storage location that Tilebox should watch. Storage locations can represent cloud buckets or local file-system locations supported by Tilebox automations.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()
locations = automations.storage_locations()
print(locations)
```

## Register the automation

Create the automation using the task prototype and storage trigger.

```python Python theme={"system"}
automations.create_storage_event_automation(
    "process-uploaded-objects",
    ProcessUploadedObject(),
    triggers=[(locations[0], "incoming/**")],
)
```

## Test and inspect

Create or change a matching object, then inspect the submitted job in the Console or through the Tilebox command-line tool.

```bash theme={"system"}
tilebox job logs <job-id>
tilebox job spans <job-id>
```

## Next steps

<Columns>
  <Card title="Storage event triggers" icon="right-to-line" href="/workflows/automations/storage-events">
    Read the full storage-event automation reference.
  </Card>

  <Card title="Debug a failed workflow run" icon="bug" href="/guides/workflows/debug-failed-run">
    Inspect task state, logs, traces, and runner context.
  </Card>
</Columns>


# Monitor workflow runs
Source: https://docs.tilebox.com/guides/operations/monitor-workflow-runs

Use the Console, CLI, and SDKs to monitor workflow jobs, task state, logs, traces, and runner behavior.

Use this guide when you need to follow workflow execution after a job is submitted. Tilebox tracks job state, task state, logs, traces, and runner context for each workflow run.

## Use the Console for live inspection

Open the [Jobs view](https://console.tilebox.com/workflows/jobs) in the Tilebox Console. Use it to inspect the task graph, task states, logs, and trace timing for a job.

The Console is the best first stop when a human is investigating a job interactively.

## Use the command-line tool for automation

Use the Tilebox command-line tool when you need structured output for scripts, CI checks, or coding agents.

```bash theme={"system"}
tilebox job logs <job-id> --json
tilebox job spans <job-id> --json
```

Ask agents to keep job IDs and command output in their working notes so they can continue from observed failures instead of guessing.

## Use SDK queries in notebooks or services

Use SDK queries when monitoring is part of a notebook, dashboard, or service.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
job = client.jobs().find("019e07b1-916b-0630-f3ba-f1c33235d174")
logs = client.jobs().query_logs(job)
spans = client.jobs().query_spans(job.id)
```

## Next steps

<Columns>
  <Card title="Inspect workflow runs" icon="eye" href="/workflows/run-and-inspect/introduction">
    Learn how Tilebox connects logs, traces, task status, and runner context.
  </Card>

  <Card title="Debug a failed workflow run" icon="bug" href="/guides/workflows/debug-failed-run">
    Work through a failed or queued job investigation.
  </Card>
</Columns>


# Query logs and spans
Source: https://docs.tilebox.com/guides/operations/query-logs-and-spans

Query workflow logs and spans from the CLI or SDKs for debugging, reports, and automated diagnostics.

Tilebox stores workflow logs and spans for each job. Query them when you need structured diagnostics outside the Console, such as in notebooks, scripts, or agent loops.

## Query from the command line

Use JSON output when another tool or agent will parse the result.

```bash theme={"system"}
tilebox job logs <job-id> --json
tilebox job spans <job-id> --json
```

Limit or sort the output during debugging.

```bash theme={"system"}
tilebox job logs <job-id> --limit 10 --sort desc
```

## Query from Python

Use the jobs client in notebooks and scripts.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
job = client.jobs().find("019e07b1-916b-0630-f3ba-f1c33235d174")

logs = client.jobs().query_logs(job)
spans = client.jobs().query_spans(job.id)

logs_df = logs.to_pandas()
spans_df = spans.to_pandas()
```

## Use the detailed reference pages

<Columns>
  <Card title="Query telemetry" icon="magnifying-glass-chart" href="/workflows/run-and-inspect/query-telemetry">
    Query logs and spans from Python and Go.
  </Card>

  <Card title="Logging" icon="rectangle-terminal" href="/workflows/run-and-inspect/logging">
    Emit structured logs from workflow tasks.
  </Card>

  <Card title="Tracing" icon="chart-gantt" href="/workflows/run-and-inspect/tracing">
    Add custom spans around important parts of task code.
  </Card>

  <Card title="Axiom integration" icon="plug" href="/workflows/run-and-inspect/integrations/axiom">
    Export workflow telemetry to Axiom.
  </Card>
</Columns>


# Iterate on Workflow Releases with Agents
Source: https://docs.tilebox.com/guides/workflows/agentic-workflow-iteration

Use a coding agent with the Tilebox command-line tool to edit, publish, deploy, run, and debug Python workflow releases.

This guide shows the recommended loop for AI-assisted workflow development. The agent edits Python workflow code, publishes a release, deploys it to a development cluster, starts a release runner, submits a test job, and inspects logs or spans before iterating.

Use this loop when you want the code under test to match the artifact that release runners execute.

## Prerequisites

Set up the Tilebox command-line tool and skills in the environment where your agent runs.

```bash theme={"system"}
curl -fsSL https://install.tilebox.com/wizard.sh | sh
export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY"
```

Ask the agent to inspect the command-line tool before changing resources.

<AgentPrompt>
  Load the Tilebox workflow skills. Inspect `tilebox agent-context workflow --output-schema` and `tilebox agent-context runner start --output-schema`. Do not modify Tilebox resources yet.
</AgentPrompt>

## Create or update the workflow project

For a new workflow project, ask the agent to initialize the project before editing task code.

<AgentPrompt>
  Inspect `tilebox agent-context workflow init --output-schema`. Then initialize a new Tilebox workflow project with `tilebox workflow init --name "<workflow-name>"`. After initialization, update the generated runner.py with the workflow tasks.
</AgentPrompt>

For an existing workflow project, ask the agent to keep the project centered on one reusable `Runner` definition.

<AgentPrompt>
  Update this Python workflow project. Keep task classes in `src/<package>/tasks.py`, export `runner = Runner(tasks=[...])` from `src/<package>/runner.py`, and keep `tilebox.workflow.toml` pointing at that runner object.
</AgentPrompt>

The direct runner path and release runner path should use the same `Runner` object. Direct execution uses `runner.connect_to(Client(), cluster=...)`; release execution uses `tilebox.workflow.toml` and `tilebox runner start`.

## Use a development cluster

Create a development cluster if you do not already have one.

```bash theme={"system"}
tilebox cluster create "workflow-dev"
```

Add the cluster slug to `tilebox.workflow.toml`.

```toml theme={"system"}
[targets.dev]
clusters = ["workflow-dev-abc123"]
```

## Build and publish a release

Ask the agent to build locally first when it needs detailed validation output.

```bash theme={"system"}
tilebox workflow build-release --debug
```

Then publish the release.

```bash theme={"system"}
tilebox workflow publish-release
```

## Deploy to the development cluster

Deploy the latest release the agent just published.

```bash theme={"system"}
tilebox workflow deploy-release --latest --target dev
```

This updates cluster deployment state. It does not submit a job.

## Start a release runner

In another terminal, start a release runner for the development cluster.

```bash theme={"system"}
tilebox runner start --cluster workflow-dev-abc123 --debug
```

Keep this runner running while the agent iterates. It can run multiple deployed releases for the cluster and reacts when the agent deploys a new release.

## Submit and inspect a test job

Submit a root task to the same cluster.

```bash theme={"system"}
tilebox job submit \
  --name my-workflow-test \
  --task tilebox.com/example/ProcessScene \
  --version v1.0 \
  --cluster workflow-dev-abc123 \
  --input '{"scene_id":"S2A_001"}' \
  --wait
```

```text theme={"system"}
Job submitted: 019ef7a2-56b4-7a86-9c4d-52a1792b1e90
```

Inspect logs and spans when the job fails or takes longer than expected.

```bash theme={"system"}
tilebox job logs <JOB_ID>
tilebox job spans <JOB_ID>
```

## Iterate safely

For compatible fixes, keep the task identifier name and major version stable. Publish and deploy the fix, then retry the failed job.

```bash theme={"system"}
tilebox workflow publish-release
tilebox workflow deploy-release --latest --target dev
tilebox job retry <JOB_ID>
```

For breaking input or behavior changes, bump the task major version and submit a new test job.


# Build and deploy a workflow project
Source: https://docs.tilebox.com/guides/workflows/build-and-deploy-workflow

Initialize a Python workflow project, publish a workflow release, and deploy it to a development cluster.

Use workflow releases when the code under test should match the code that runners execute. A release packages a Python workflow project, records the discovered task identifiers, and lets release runners load the deployed code from a cluster.

This guide summarizes the build and deploy path. For the underlying model, see [Workflow releases](/workflows/concepts/workflow-releases) and [Release lifecycle](/workflows/build-and-deploy/releases).

## Prerequisites

* You have installed the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli).
* `TILEBOX_API_KEY` is set in the shell where you run commands.
* `uv` is installed and available on `PATH`.

## Initialize the project

Create a project directory and initialize it as a Tilebox workflow project.

```bash theme={"system"}
mkdir my-workflow
cd my-workflow
tilebox workflow init --name "My Workflow"
```

The init command creates the Tilebox workflow, writes `tilebox.workflow.toml`, creates a minimal Python project with `runner.py`, adds the `tilebox` dependency, and runs `uv sync`. It aborts if `tilebox.workflow.toml`, `pyproject.toml`, `runner.py`, or `uv.lock` already exists in the current directory.

<Tree>
  <Tree.Folder name="my-workflow">
    <Tree.File name="pyproject.toml" />

    <Tree.File name="uv.lock" />

    <Tree.File name="tilebox.workflow.toml" />

    <Tree.File name="runner.py" />
  </Tree.Folder>
</Tree>

## Configure a deployment target

Create a logical cluster to deploy the workflow release to, enabling [release runners](/workflows/concepts/runners#release-runners).

```bash theme={"system"}
tilebox cluster create "workflow-dev"
```

```text Output theme={"system"}
Slug               workflow-dev-<...>
Name               workflow-dev
Deletable          true
```

Add the returned cluster slug to `tilebox.workflow.toml` so deployment commands can refer to a logical environment instead of repeating cluster slugs.

```toml tilebox.workflow.toml theme={"system"}
[workflow]
slug = "my-workflow-<...>"
root = "."
runner = "runner:runner"

[build]
include = [
  "tilebox.workflow.toml",
  "pyproject.toml",
  "uv.lock",
  "runner.py",
]

[targets.dev]
clusters = ["workflow-dev-<...>"]
```

## Build and publish the release

Create a new release of the workflow and publish it to Tilebox.

```bash theme={"system"}
tilebox workflow publish-release
```

Publishing creates an immutable release. It does not change what any cluster runs until you deploy it.

<Tip>
  Build locally first when you want detailed validation output.

  `tilebox workflow build-release --debug`
</Tip>

## Deploy to a development cluster

Deploy the release to the development target.

```bash theme={"system"}
tilebox workflow deploy-release --latest --target dev
```

Start a release runner in an environment you control.

```bash theme={"system"}
tilebox runner start --cluster workflow-dev-9xK2mQ4pL8nR7s --debug
```

## Run and inspect a job

Submit a task that matches one of the identifiers discovered from the release.

```bash theme={"system"}
tilebox job submit \
  --name my-workflow-test \
  --task my-workflow/HelloWorld \
  --version v0.1 \
  --cluster workflow-dev-9xK2mQ4pL8nR7s \
  --input '{"name":"Tilebox"}' \
  --wait
```

```text theme={"system"}
Job submitted: 019ef7a2-56b4-7a86-9c4d-52a1792b1e90
```

Inspect the result before making the next change.

```bash theme={"system"}
tilebox job logs <JOB_ID>
tilebox job spans <JOB_ID>
```

## Next steps

<Columns>
  <Card title="Project structure" icon="folder-tree" href="/workflows/build-and-deploy/project-structure">
    Learn the details of Python workflow project layout.
  </Card>

  <Card title="Deploy to your compute" icon="server" href="/guides/workflows/deploy-to-your-compute">
    Choose where release runners run and how clusters map to environments.
  </Card>
</Columns>


# Debug a failed workflow run
Source: https://docs.tilebox.com/guides/workflows/debug-failed-run

Inspect task state, logs, traces, runner context, and cluster alignment when a workflow job fails or stays queued.

Use this guide when a workflow job fails, runs slower than expected, or stays queued. Tilebox records job state, task state, logs, traces, and runner context so you can identify whether the problem is in task code, task routing, dependencies, or the runner environment.

## Find the job

Open the job in the [Tilebox Console](https://console.tilebox.com/workflows/jobs), or use the Tilebox command-line tool if you already have the job ID.

```bash theme={"system"}
tilebox job logs <job-id>
tilebox job spans <job-id>
```

## Check task state first

Start with the task graph. A failed task often points to task code or runtime dependencies. A queued task often points to cluster, runner, or task registration mismatch.

Common checks:

* The job was submitted to the intended cluster.
* A runner is connected to the same cluster.
* The runner advertises the submitted task identifier and compatible version.
* Any task dependencies are complete.
* Retry limits have not been exhausted.

## Inspect logs

Logs show messages emitted by task code and runner context attached by Tilebox.

```bash theme={"system"}
tilebox job logs <job-id>
```

Use structured log fields in your tasks so the relevant scene ID, product ID, path, or model name appears in the log record.

## Inspect traces

Traces show task timing, parent-child relationships, custom spans, and failures.

```bash theme={"system"}
tilebox job spans <job-id>
```

Use traces to find slow subtasks, repeated retries, and failures inside a specific custom span.

## Fix and rerun

For direct runners, fix the code and restart the runner process. For release runners, publish a fixed release and deploy it.

If the fix is compatible with the failed task input schema and task major version, retry the job. If the change is breaking, submit a new job with a new task version.

<Columns>
  <Card title="Retry with a compatible release" icon="rotate" href="/guides/workflows/retry-with-compatible-release">
    Publish a compatible fix, deploy it to the same cluster, and retry failed work.
  </Card>

  <Card title="Inspect workflow runs" icon="eye" href="/workflows/run-and-inspect/introduction">
    Learn how logs, traces, task status, and runner context fit together.
  </Card>
</Columns>


# Deploy to your compute
Source: https://docs.tilebox.com/guides/workflows/deploy-to-your-compute

Run Tilebox workflow releases on compute environments you control by using clusters and release runners.

Tilebox manages workflow state, releases, deployments, jobs, logs, and traces. Your compute environment runs the runner process that executes the work. This lets workflows run on local machines, cloud virtual machines, Kubernetes clusters, on-premises systems, or controlled customer infrastructure.

## Choose a cluster for the environment

A cluster is the routing boundary for jobs and runners. Jobs submitted to a cluster can only be claimed by runners connected to the same cluster.

Use separate clusters for environments that should not run the same code by accident, such as development and production-like compute.

```bash theme={"system"}
tilebox cluster create "workflow-dev"
tilebox cluster create "workflow-prod"
```

## Deploy a release to the cluster

Deploy a published workflow release to the cluster or to a target defined in `tilebox.workflow.toml`.

```bash theme={"system"}
tilebox workflow deploy-release --latest --cluster workflow-dev
```

For repeated deployments, define targets in the workflow configuration.

```toml theme={"system"}
[targets.dev]
clusters = ["workflow-dev"]

[targets.production]
clusters = ["workflow-prod"]
```

Then deploy by target name.

```bash theme={"system"}
tilebox workflow deploy-release --latest --target dev
```

## Start release runners where the work should run

Start one or more release runners in the environment that has the required network access, credentials, hardware, and data access.

```bash theme={"system"}
tilebox runner start --cluster workflow-dev --debug
```

The runner watches its cluster, downloads missing release artifacts, starts the workflow runtime, and advertises the tasks it can execute. Updating a deployment changes what the runner can execute without rebuilding the runner process.

## Bundle the release runner in a container

For cloud or Kubernetes deployments, package the release runner into a small container image. The image only needs Python, `uv`, the Tilebox command-line tool, and any system dependencies your workflow runtime needs. The workflow code itself comes from the deployed workflow release.

```dockerfile Dockerfile theme={"system"}
FROM python:3.13-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

# Install system dependencies.
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git git-lfs openssh-client \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

RUN curl -fsSL https://install.tilebox.com/cli.sh | TILEBOX_INSTALL_DIR=/usr/local/bin TILEBOX_NO_INSTALL_COMPLETIONS=1 sh

# Required at runtime: set TILEBOX_CLUSTER to a valid cluster slug and
# TILEBOX_API_KEY to an API key that can read deployments and claim tasks.
ENV TILEBOX_CLUSTER=""
ENV TILEBOX_API_KEY=""

CMD ["tilebox", "runner", "start"]
```

Build and publish this image with your normal container workflow. At runtime, provide `TILEBOX_CLUSTER` and `TILEBOX_API_KEY` through your deployment system rather than baking secrets into the image.

In Kubernetes, run the image as a `Deployment` and store `TILEBOX_API_KEY` in a `Secret`. Set `TILEBOX_CLUSTER` through the pod environment and scale replicas to increase runner concurrency. In Google Cloud, run the same image on Cloud Run jobs, GKE, or Compute Engine depending on your workload constraints. In AWS, run it on ECS, EKS, or EC2 and inject the API key through your secret manager or task definition.

## Scale runner processes

Scale the number of runner containers or virtual machine instances when you want more parallelism. In Kubernetes, increase the `Deployment` replica count. In GCP or AWS, use the scaling controls of the service that runs the container, such as Cloud Run, GKE, ECS, EKS, or an auto-scaling VM group. Each runner process connects to the same cluster and claims compatible tasks independently.

As an alternative for local testing or constrained environments, you can run multiple runner processes inside one container or shell session. Use `tilebox parallel` only for that case.

```bash theme={"system"}
tilebox parallel -n 4 -- tilebox runner start --cluster workflow-dev
```

Tilebox does not require the runner process to run in Tilebox-managed infrastructure. Use the process manager, scheduler, or container platform that fits your compute environment.

## Verify cluster alignment

If a job stays queued, check that these three values match:

1. The job was submitted to the expected cluster.
2. The workflow release is deployed to that cluster.
3. A release runner is running for that cluster.

For details, see [Cluster deployments](/workflows/build-and-deploy/cluster-deployments) and [Runners](/workflows/concepts/runners).


# Execute tasks in parallel
Source: https://docs.tilebox.com/guides/workflows/execute-tasks-in-parallel

Submit multiple workflow subtasks and process them faster by running multiple direct runners at the same time.

Use this guide when a workflow can split work into independent tasks. You will create a small workflow that submits 20 sleep subtasks, submit one job, run it with one direct runner, and then run the same workflow with five direct runners in parallel.

The example is intentionally simple. `time.sleep` stands in for real work such as downloading scenes, processing tiles, calling a model, or writing output files.

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed [`uv`](https://docs.astral.sh/uv/).
* You have installed the [Tilebox command-line tool](/agents-and-ai-tools/tilebox-cli) if you want to use `tilebox parallel`.

```bash theme={"system"}
export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY"
```

## Create the workflow file

Create a file named `parallel_workflow.py`. The script uses inline `uv` dependencies, so you can run it directly with `uv run parallel_workflow.py`.

```python parallel_workflow.py theme={"system"}
# /// script
# dependencies = ["cyclopts", "tilebox"]
# ///

import time

from cyclopts import App
from tilebox.workflows import Client, ExecutionContext, Runner, Task


app = App()


class ParallelSleepWorkflow(Task):
    count: int
    seconds: float

    def execute(self, context: ExecutionContext) -> None:
        context.logger.info(
            "Submitting sleep subtasks",
            count=self.count,
            seconds=self.seconds,
        )
        context.submit_subtasks(
            [
                SleepTask(index=index, seconds=self.seconds)
                for index in range(self.count)
            ]
        )


class SleepTask(Task):
    index: int
    seconds: float

    def execute(self, context: ExecutionContext) -> None:
        context.current_task.display = f"SleepTask({self.index})"
        context.logger.info("Starting sleep task", index=self.index)
        time.sleep(self.seconds)
        context.logger.info("Finished sleep task", index=self.index)


runner = Runner(tasks=[ParallelSleepWorkflow, SleepTask])


def submit_job(count: int, seconds: float) -> None:
    client = Client()
    job = client.jobs().submit(
        "parallel-sleep-workflow",
        ParallelSleepWorkflow(count=count, seconds=seconds),
    )
    print(f"Submitted job: {job.id}")
    print(f"Open in Console: https://console.tilebox.com/workflows/jobs/{job.id}")


def run_runner() -> None:
    client = Client()
    runner.connect_to(client).run_all()


@app.default
def main(submit: bool = False, count: int = 20, seconds: float = 5.0) -> None:
    """Run a direct runner, or submit a new job with --submit."""
    if submit:
        submit_job(count=count, seconds=seconds)
        return

    run_runner()


if __name__ == "__main__":
    app()
```

The root task, `ParallelSleepWorkflow`, does not do the slow work itself. It submits many independent `SleepTask` subtasks. Tilebox tracks the tasks in one job and lets any eligible runner claim queued work.

## Submit a job

Submit one job with 20 subtasks. The script exits after submitting the job, so no work is executed yet.

```bash theme={"system"}
uv run parallel_workflow.py --submit --count 20 --seconds 5
```

Copy the job ID from the output. You can inspect it in the Console while runners process the queue.

```plaintext Output theme={"system"}
Submitted job: 019f2c8c-3df2-4ed0-9d8f-8a4f19c47a7c
Open in Console: https://console.tilebox.com/workflows/jobs/
```

## Run one direct runner

Start one direct runner from the same file.

```bash theme={"system"}
uv run parallel_workflow.py
```

The runner executes tasks, but only one after the other, and exits when no more work is available. With one runner, the sleep subtasks don't run in parallel at all.

## Run five direct runners

Submit another job, then start five runner processes for the same workflow file.

```bash theme={"system"}
uv run parallel_workflow.py --submit --count 20 --seconds 5
tilebox parallel -n 5 -- uv run parallel_workflow.py
```

This starts five direct runners. Each process registers the same task classes and asks Tilebox for work. Tilebox assigns queued tasks across the available runners, so multiple `SleepTask` instances run at the same time.

<Tip>
  Takeaway: use `tilebox parallel -n 5 -- uv run parallel_workflow.py` to start five local direct runners for the same workflow file.
</Tip>

## What to expect

The first runner to claim `ParallelSleepWorkflow` submits the subtasks. After that, all runners can claim compatible `SleepTask` tasks from the same job.

In the Console, you should see:

* one root task that submits the subtask fan-out
* many `SleepTask(index)` tasks
* multiple tasks running at overlapping times when five runners are active
* logs from each task attached to the same job

For command-line inspection, query logs or spans for the job:

```bash theme={"system"}
tilebox job logs <job-id>
tilebox job spans <job-id>
```

## Next steps

<Columns>
  <Card title="Runners" icon="list-check" href="/workflows/concepts/runners">
    Learn how runners claim queued tasks and how direct runners differ from release runners.
  </Card>

  <Card title="Tasks" icon="laptop-code" href="/workflows/concepts/tasks">
    Learn how parent tasks submit subtasks, define dependencies, and report progress.
  </Card>

  <Card title="Debug a failed workflow run" icon="bug" href="/guides/workflows/debug-failed-run">
    Inspect task state, logs, traces, runner context, and cluster alignment.
  </Card>
</Columns>


# Multi-language Workflows
Source: https://docs.tilebox.com/guides/workflows/multi-language

Combine tasks written in different programming languages within a single workflow, letting you choose the best language for each individual processing step.

<Columns>
  <Card title="Open in GitHub" icon="github" href="https://github.com/tilebox/examples/tree/main/workflows-multilang">
    The code for this guide is available on GitHub.
  </Card>
</Columns>

## Tilebox languages and SDKs

Tilebox supports multiple languages and SDKs for running workflows.
All Tilebox SDKs and workflows are designed to be interoperable, which means it's possible to have a workflow where individual tasks are executed in different languages.
Check out [Languages & SDKs](/sdks/introduction) to learn more about currently available programming SDKs.

## Why multi-language workflows?

You might need to use multiple languages in a single workflow for many reasons, such as:

* You want to use a language that is better suited for a specific task (for example Python for data processing, Go for a backend API)
* You want to use a library that is only available in a specific language (for example xarray in Python)
* You started prototyping in Python, but need to start migrating the compute-intensive parts of your workflow to a different language for performance reasons

## Multi-language workflow example

This guide will tackle the first use case: you have a tasking server in Go and want to offload some of the processing to Python.

## Defining tasks in Python and Go

```python Python theme={"system"}
class ScheduleImageCapture(Task):
    # The input parameters must match the ones defined in the Go task
    location: tuple[float, float]  # lat_lon
    resolution_m: int
    spectral_bands: list[float]  # spectral bands in nm

    def execute(self, context: ExecutionContext) -> None:
        # Here you can implement your task logic, submit subtasks, etc.
        context.logger.info(
            "Image captured",
            location=self.location,
            resolution_m=self.resolution_m,
            spectral_bands=self.spectral_bands,
        )

    @staticmethod
    def identifier() -> tuple[str, str]:
        # The identifier must match the one defined in the Go task
        return "tilebox.com/schedule_image_capture", "v1.0"
```

```go Go theme={"system"}
type ScheduleImageCapture struct {
    // json tags must match the Python task definition
    Location      [2]float64 `json:"location"` // lat_lon
    ResolutionM   int        `json:"resolution_m"`
    SpectralBands []float64  `json:"spectral_bands"` // spectral bands in nm
}

// No need to define the Execute method since we're only submitting the task

// Identifier must match with the task identifier in the Python runner
func (t *ScheduleImageCapture) Identifier() workflows.TaskIdentifier {
    return workflows.NewTaskIdentifier("tilebox.com/schedule_image_capture", "v1.0")
}
```

A couple important points to note:

<AccordionGroup>
  <Accordion title="Input parameters">
    The dataclass parameters in Python must match the struct fields in Go, including the types and the names (through the JSON tags in Go).

    Due to Go and Python having different naming conventions, it's recommended to use JSON tags in the Go struct to match the Python dataclass field names to respect the language-specific conventions.

    Go fields must start with an uppercase letter to be serialized to JSON.

    <Note>
      The need for JSON tags in the preceding Go code is currently necessary but might be removed in the future.
    </Note>
  </Accordion>

  <Accordion title="Execute method">
    The execute method is defined in the Python task but not in the Go task since Go will only be used to submit the task, not executing it.
  </Accordion>

  <Accordion title="Identifier">
    It's necessary to define the `identifier` method in both the Python and Go tasks and to make sure they match.
    The `identifier` method has two values, the first being an arbitrary unique task identifier and the second being the version in the `v{major}.{minor}` format.
  </Accordion>
</AccordionGroup>

## Creating a Go server that submits jobs

Write a simple HTTP tasking server in Go with a `/submit` endpoint that accepts requests to submit a `ScheduleImageCapture` job.

<Note>
  Both Go and Python code are using `test-cluster-tZD9Ca2qsqt4V` as the cluster slug. You should replace it with your own cluster slug, which you can create in the [Tilebox Console](https://console.tilebox.com/workflows/clusters).
</Note>

```go Go theme={"system"}
func main() {
    log.Println("Server starting on http://localhost:8080")

    client := workflows.NewClient()
    http.HandleFunc("/submit", submitHandler(client))
    log.Fatal(http.ListenAndServe(":8080", nil))
}

// Submit a job based on some query parameters
func submitHandler(client *workflows.Client) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        latArg := r.URL.Query().Get("lat")
        lonArg := r.URL.Query().Get("lon")
        resolutionArg := r.URL.Query().Get("resolution")
        bandsArg := r.URL.Query().Get("bands[]")

        latFloat, err := strconv.ParseFloat(latArg, 64)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }
        lonFloat, err := strconv.ParseFloat(lonArg, 64)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }

        resolutionM, err := strconv.Atoi(resolutionArg)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }

        var spectralBands []float64
        for _, bandArg := range strings.Split(bandsArg, ",") {
            band, err := strconv.ParseFloat(bandArg, 64)
            if err != nil {
                http.Error(w, err.Error(), http.StatusBadRequest)
                return
            }
            spectralBands = append(spectralBands, band)
        }

        job, err := client.Jobs.Submit(r.Context(), "Schedule Image capture",
            []workflows.Task{
                &ScheduleImageCapture{
                    Location:      [2]float64{latFloat, lonFloat},
                    ResolutionM:   resolutionM,
                    SpectralBands: spectralBands,
                },
            },
        )
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        _, _ = io.WriteString(w, fmt.Sprintf("Job submitted: %s\n", job.ID))
    }
}
```

<Note>
  In the same way that you can submit jobs across languages you can also submit subtasks across languages.
</Note>

## Creating a Python runner

Write a Python script that starts a runner and registers the `ScheduleImageCapture` task.

```python Python theme={"system"}
from tilebox.workflows import Client

def main():
    client = Client()
    runner = client.runner(tasks=[ScheduleImageCapture])
    runner.run_forever()

if __name__ == "__main__":
    main()
```

## Testing it

Start the Go server.

```bash Shell theme={"system"}
go run .
```

In another terminal, start the Python runner.

```bash Shell theme={"system"}
uv run runner.py
```

Submit a job to the Go server.

```bash Shell theme={"system"}
curl http://localhost:8080/submit?lat=40.75&lon=-73.98&resolution=30&bands[]=489.0,560.6,666.5
```

Check the Python runner logs for the following entry:

```plaintext Logs theme={"system"}
Image captured location=[40.75, -73.98] resolution_m=30 spectral_bands=[489, 560.6, 666.5]
```

## Next Steps

<Columns>
  <Card title="Open in GitHub" icon="github" href="https://github.com/tilebox/examples/tree/main/workflows-multilang">
    The code for this guide is available on GitHub.
  </Card>
</Columns>

As a learning exercise, you can try to change the [News API Workflow](/workflows/concepts/tasks#dependencies-example) to replace the `FetchNews` task with a Go task and keep all the other tasks in Python.
You'll learn how to submit a subtask in another language than what the current task is executed in.


# Retry a failed job after a compatible fix
Source: https://docs.tilebox.com/guides/workflows/retry-with-compatible-release

Publish a fixed workflow release, deploy it to the same cluster, and retry failed work without resubmitting the whole job.

When a workflow job fails because of a bug in task code, you can often fix the code, publish a compatible release, deploy it to the same cluster, and retry the failed job. Tilebox resumes from failed tasks instead of rerunning completed work.

## Confirm the fix is compatible

A retry can use the fixed release when the failed task and the new task registration are compatible.

Keep these stable:

* task identifier name
* task major version
* task input schema

Use a new major task version and submit a new job when the input schema or behavior is no longer compatible with the failed task.

## Publish the fixed release

After editing the workflow code, publish a new release.

```bash theme={"system"}
tilebox workflow publish-release
```

For validation details, build locally first.

```bash theme={"system"}
tilebox workflow build-release --debug
```

## Deploy to the same cluster

Deploy the fixed release to the same cluster that received the original job.

```bash theme={"system"}
tilebox workflow deploy-release --latest --cluster workflow-dev
```

The job cluster, release deployment cluster, and release runner cluster must match.

## Retry the job

Retry the failed job.

```bash theme={"system"}
tilebox job retry <job-id>
```

Then inspect logs and spans to confirm the fixed task completed.

```bash theme={"system"}
tilebox job logs <job-id>
tilebox job spans <job-id>
```

## Next steps

<Columns>
  <Card title="Workflow releases" icon="box-open" href="/workflows/concepts/workflow-releases">
    Understand release compatibility and task registrations.
  </Card>

  <Card title="Debug a failed workflow run" icon="bug" href="/guides/workflows/debug-failed-run">
    Inspect task state, logs, traces, and runner context.
  </Card>
</Columns>


# Run your first workflow
Source: https://docs.tilebox.com/guides/workflows/run-your-first-workflow

Define a small workflow task, start a direct runner, submit a job, and inspect the result.

This guide runs a minimal workflow with a direct runner. It gives you the fastest path to the Tilebox Workflows execution model before publishing workflow releases or deploying to a cluster.

Use this path when you are developing locally as a developer. If you want an agent to do the setup and iteration for you, start with [Onboard your agent](/onboard-your-agent) and then follow [Iterate on workflow releases with agents](/guides/workflows/agentic-workflow-iteration).

## Prerequisites

* You have a [Tilebox API key](/authentication).
* You have installed the [Python SDK](/sdks/python/install).

```bash theme={"system"}
uv add tilebox
export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY"
```

## Create a workflow file

Create a file named `hello_workflow.py`.

```python hello_workflow.py theme={"system"}
from tilebox.workflows import Client, ExecutionContext, Runner, Task


class HelloWorkflow(Task):
    name: str

    def execute(self, context: ExecutionContext) -> None:
        context.logger.info("Running root task", name=self.name)
        context.submit_subtask(WriteGreeting(name=self.name))


class WriteGreeting(Task):
    name: str

    def execute(self, context: ExecutionContext) -> None:
        context.logger.info("Hello from Tilebox", name=self.name)


if __name__ == "__main__":
    client = Client()
    runner = Runner(tasks=[HelloWorkflow, WriteGreeting])

    job = client.jobs().submit(
        "hello-workflow",
        HelloWorkflow(name="Earth"),
    )
    print(f"Submitted job: {job.id}")

    runner.connect_to(client).run_all()
```

The root task submits one subtask. Tilebox tracks both tasks as part of the same job, and the direct runner executes all eligible work.

## Run the workflow

Run the file locally.

```bash theme={"system"}
uv run python hello_workflow.py
```

The script submits a job, starts a direct runner, executes the root task and subtask, and exits after the queued work is complete.

## Inspect the job

Open the job in the [Tilebox Console](https://console.tilebox.com/workflows/jobs) to inspect task state, logs, and the execution trace.

For scripted inspection, use the jobs client or the Tilebox command-line tool:

```bash theme={"system"}
tilebox job logs <job-id>
tilebox job spans <job-id>
```

## Next steps

<Columns>
  <Card title="Tasks" icon="laptop-code" href="/workflows/concepts/tasks">
    Define task inputs, subtasks, dependencies, retries, and identifiers.
  </Card>

  <Card title="Build and deploy a workflow" icon="rocket" href="/guides/workflows/build-and-deploy-workflow">
    Package the same kind of workflow code as a release and deploy it to a cluster.
  </Card>
</Columns>


# Onboard your agent
Source: https://docs.tilebox.com/onboard-your-agent

Configure an AI agent with the Tilebox CLI and Tilebox skills.

<div>
  This page is intended for users to guide them with configuring their agent with Tilebox.
  Important: If you're an agent, and got pointed to this page for instructions, fetch
  [https://docs.tilebox.com/agent-onboarding.md](https://docs.tilebox.com/agent-onboarding.md) instead. Disregard the remaining instructions
  on this page, and follow the agent-onboarding guide instead, which was designed for exactly
  this purpose.
</div>

Onboard your AI agent to Tilebox by installing the Tilebox CLI and Agent Skills, then configuring your API key. After completing it, your agent will be able to manage your Tilebox resources and use Tilebox-specific skills to create and query datasets, submit and monitor jobs.
It will also be able to write geospatial workflows, deploy it to clusters, submit jobs and iterate on the workflow code after inspecting job logs and checking outputs.
If you want to use Tilebox directly as a developer, start with the [developer quickstart](/quickstart).

<Tip>
  **Using a LLM Chat interface instead of an agent?**
  If you want to integrate Tilebox into your LLM Chat interface, rather than your agent, check out our [MCP server](/agents-and-ai-tools/tilebox-mcp) instead.
</Tip>

<Steps>
  <Step title="Install the Tilebox CLI and Agent Skills">
    Install the Tilebox CLI and Tilebox Agent Skills with the setup wizard.

    ```bash theme={"system"}
    curl -fsSL https://install.tilebox.com/wizard.sh | sh
    ```

    The CLI is the default interface for coding agents because it works in the same terminal where the agent edits files, runs tests, and applies changes. The skills add task-level instructions for CLI usage, dataset management, job monitoring, workflow authoring, and automations.
  </Step>

  <Step title="Authenticate the CLI">
    Set `TILEBOX_API_KEY` in the environment where your agent runs. You can create an API key in the [Tilebox Console](https://console.tilebox.com/settings/api-keys).

    ```bash theme={"system"}
    export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY"
    ```

    Use a key with the smallest permissions needed for the task you want the agent to perform.
  </Step>

  <Step title="Verify the setup">
    Ask your agent to inspect its Tilebox capabilities before assigning a larger task.

    <AgentPrompt>
      Check your Tilebox setup. Confirm whether the `tilebox` CLI is installed, whether you can inspect commands with `tilebox agent-context`, which Tilebox skills are available, and which datasets are available. Do not modify any Tilebox resources.
    </AgentPrompt>
  </Step>
</Steps>

## Next steps

Learn how to [Iterate on Workflow Releases with Agents](/guides/workflows/agentic-workflow-iteration), or check out the [CLI Reference](/agents-and-ai-tools/tilebox-cli) to get an overview of useful `tilebox` terminal commands.


# Quickstart
Source: https://docs.tilebox.com/quickstart

Start here as a developer to create an API key, query Earth data, and run a small workflow.

This quickstart is for developers using Tilebox directly from a terminal, notebook, or SDK. If you want an AI coding agent to work with Tilebox for you, start with [Onboard your agent](/onboard-your-agent).

You will create an API key, query open Sentinel-2 metadata, and run a small workflow task using the Tilebox [Python SDK](/sdks/python).

## Start in a Notebook

Explore the provided [Sample Notebooks](/sdks/python/sample-notebooks) to begin your journey with Tilebox. These notebooks offer a step-by-step guide to using the API and showcase many features supported by Tilebox Python clients. You can also use these notebooks as a foundation for your own projects.

## Start on Your Device

If you prefer to work locally, follow these steps to get started.

<Steps>
  <Step title="Create an API Key">
    Create an API key by logging into the [Tilebox Console](https://console.tilebox.com), navigating to [Settings -> API Keys](https://console.tilebox.com/settings/api-keys), and clicking the "Create API Key" button.

    Then, add it to your environment

    ```bash theme={"system"}
    export TILEBOX_API_KEY=<your-api-key>
    ```
  </Step>

  <Step title="Choose your working environment">
    Tilebox can be used from the browser, terminal, or via our SDKs running locally or in interactive notebook environments. This quickstart guides you through setting up the Tilebox **Python SDK locally**.
    Alternatively, you can also check out any of the following ways of using Tilebox.

    <CardGroup>
      <Card title="Console" icon="display" href="/console" />

      <Card title="Command line" icon="terminal" href="/agents-and-ai-tools/tilebox-cli" />

      <Card title="Go SDK" icon="golang" href="/sdks/go" />

      <Card title="Agents" icon="robot" href="/onboard-your-agent" />

      <Card title="Notebooks" icon="book-open" href="/sdks/python/sample-notebooks" />
    </CardGroup>

    To install the python SDK locally, run the following command.

    ```bash theme={"system"}
    uv add tilebox
    ```
  </Step>

  <Step title="Query Data">
    Use the datasets client to query data from a dataset.

    ```python Python theme={"system"}
    from tilebox.datasets import Client
    from shapely import Polygon

    # define a search area (optional)
    new_york_city = Polygon(
        [(-74.40, 41.02), (-74.48, 40.27), (-73.37, 40.34),
        (-73.39, 41.05), (-74.40, 41.02)]
    )

    client = Client(token="YOUR_TILEBOX_API_KEY")

    # select a dataset
    dataset = client.dataset("open_data.copernicus.sentinel2_msi")
    # and query data
    scenes_new_york_january_2026 = dataset.query(
      collections=["S2A_S2MSI2A", "S2B_S2MSI2A", "S2C_S2MSI2A"],
      temporal_extent=("2026-01-01", "2026-02-01"),
      spatial_extent=new_york_city
    )
    print(scenes_new_york_january_2026.granule_name)
    ```
  </Step>

  <Step title="Run a Workflow">
    Use the workflows client to create a task and submit it as a job.

    ```python Python theme={"system"}
    from tilebox.workflows import Client, Runner, Task
    from tilebox.workflows.observability.logging import configure_console_logging

    configure_console_logging()

    # Replace with your actual token
    client = Client(token="YOUR_TILEBOX_API_KEY")

    class HelloWorldTask(Task):
        greeting: str = "Hello"
        name: str = "World"

        def execute(self, context):
            context.logger.info(f"{self.greeting} {self.name}, from the main task!")
            context.submit_subtask(HelloSubtask(name=self.name))

    class HelloSubtask(Task):
        name: str

        def execute(self, context):
            context.logger.info(f"Hello from the subtask!", name=self.name)

    # Initiate the job
    jobs = client.jobs()
    job = jobs.submit("parameterized-hello-world", HelloWorldTask(greeting="Greetings", name="Universe"))

    # Run the tasks
    runner = Runner(tasks=[HelloWorldTask, HelloSubtask])
    runner.connect_to(client).run_all()

    print("Explore the job you just submitted in the Tilebox Console.")
    print("Check out tasks, log messages, and execution timining:")
    print(f"https://console.tilebox.com/workflows/jobs/{job.id}")
    ```
  </Step>

  <Step title="Explore Further">
    Review the following guides to learn more about the modules that make up Tilebox:

    <Columns>
      <Card title="Build a spatio-temporal catalog" icon="database" href="/guides/datasets/build-spatiotemporal-catalog">
        Learn how to create a custom dataset catalog with the Python SDK.
      </Card>

      <Card title="Ingest into a spatio-temporal catalog" icon="up-from-bracket" href="/guides/datasets/ingest-into-spatiotemporal-catalog">
        Learn how to ingest GeoParquet metadata into an existing spatio-temporal catalog.
      </Card>

      <Card title="Debug a failed workflow run" icon="bug" href="/guides/workflows/debug-failed-run">
        Inspect task state, logs and traces when a workflow job fails.
      </Card>

      <Card title="Build and deploy a workflow project" icon="rocket" href="/guides/workflows/build-and-deploy-workflow">
        Package a Python workflow project, publish a release, and deploy it to a cluster.
      </Card>

      <Card title="Agentic workflow iteration" icon="robot" href="/guides/workflows/agentic-workflow-iteration">
        Use a coding agent with the Tilebox CLI to build, deploy, run, and debug workflow releases.
      </Card>
    </Columns>
  </Step>
</Steps>


# Examples
Source: https://docs.tilebox.com/sdks/go/examples

Get started quickly with standalone, runnable examples that demonstrate common patterns for building workflows and accessing datasets with the Go SDK.

To quickly become familiar with the Go client, you can explore some standalone examples.

You can access the examples on [<Icon icon="github" /> GitHub](https://github.com/tilebox/tilebox-go/tree/main/examples).
More examples can be found throughout the docs.

## Workflows examples

<Columns>
  <Card title="Hello World" icon="hand-wave" href="https://github.com/tilebox/tilebox-go/tree/main/examples/workflows/helloworld">
    How to use Tilebox Workflows to submit and execute a simple task.
  </Card>

  <Card title="Protobuf tasks" icon="file-code" href="https://github.com/tilebox/tilebox-go/blob/main/examples/workflows/protobuf-task">
    How to submit a task and run a workflow using protobuf messages.
  </Card>

  <Card title="Axiom Observability" icon="chart-gantt" href="https://github.com/tilebox/tilebox-go/tree/main/examples/workflows/axiom">
    How to set up tracing and logging for workflows using Axiom observability platform.
  </Card>

  <Card title="OpenTelemetry Observability" icon="eye" href="https://github.com/tilebox/tilebox-go/blob/main/examples/workflows/opentelemetry">
    How to set up tracing and logging for workflows using OpenTelemetry.
  </Card>
</Columns>

## Datasets examples

<Columns>
  <Card title="Query" icon="server" href="https://github.com/tilebox/tilebox-go/blob/main/examples/datasets/query/main.go">
    How to query datapoints from a Tilebox dataset.
  </Card>

  <Card title="Ingest and delete" icon="up-from-bracket" href="https://github.com/tilebox/tilebox-go/blob/main/examples/datasets/ingest/main.go">
    How to create a collection, ingest datapoints, and then delete them.
  </Card>
</Columns>


# Installation
Source: https://docs.tilebox.com/sdks/go/install

Install and set up the Tilebox Go SDK and its companion code generation tool to start building workflows and accessing datasets from your applications.

## Package Overview

Tilebox offers a Go SDK for accessing Tilebox services. The Tilebox command-line tool includes `tilebox dataset generate`, which generates Go protobuf types for Tilebox datasets.

<Columns>
  <Card title="tilebox-go" icon="golang" href="https://github.com/tilebox/tilebox-go">
    Datasets and workflows client for Tilebox
  </Card>

  <Card title="Tilebox CLI" icon="terminal" href="/agents-and-ai-tools/tilebox-cli">
    Command-line tools to query datasets, run workflows, and generate Go dataset types
  </Card>
</Columns>

## Installation

Add `tilebox-go` to your project.

```bash Shell theme={"system"}
go get github.com/tilebox/tilebox-go
```

Install the Tilebox command-line tool on your machine.

```bash Shell theme={"system"}
curl -fsSL https://install.tilebox.com/cli.sh | sh
```


# Protobuf
Source: https://docs.tilebox.com/sdks/go/protobuf

Understand how Protocol Buffers provide strongly typed data structures for working with Tilebox datasets in Go, and how to generate them for your projects.

Tilebox uses [Protocol Buffers](https://protobuf.dev/), with a custom generation tool, combined with standard Go data structures.

[Protocol Buffers](https://protobuf.dev/) (often referred to as `protobuf`) is a schema definition language with an efficient binary format and native language support for lots of languages, including Go.
Protocol buffers are open source since 2008 and are maintained by Google.

## Generate Go types

Protobuf schemas are typically defined in a `.proto` file, and then converted to a native Go struct using the protobuf compiler.
Tilebox datasets already define a protobuf schema as well. Use `tilebox dataset generate` to generate Go structs for existing datasets.

See [Installation](/sdks/go/install) for more details on how to install the Tilebox command-line tool.

```sh theme={"system"}
tilebox dataset generate --slug open_data.copernicus.sentinel1_sar
```

The preceding command will generate a `./protogen/tilebox/v1/sentinel1_sar.pb.go` file. Use `--out`, `--package`, and `--name` to change the default output directory, protobuf package, or message name.

This file contains everything needed to work with the [Sentinel-1 SAR](https://console.tilebox.com/datasets/explorer/e27e6a58-c149-4379-9fdf-9d43903cba74) dataset.
It's recommended to check the generated files you use in your version control system.

If you open this file, you will see that it starts with `// Code generated by protoc-gen-go. DO NOT EDIT.`.
It means that the file was generated by the `protoc-gen-go` tool, which is part of the protobuf compiler.
After editing a dataset, you can call the generate command again to ensure that the changes are reflected in the generated file.

The file contains a `Sentinel1Sar` struct, which is a Go struct that represents a datapoint in the dataset.

```go Go theme={"system"}
type Sentinel1Sar struct {
  xxx_hidden_GranuleName         *string                `protobuf:"bytes,1,opt,name=granule_name,json=granuleName"`
  xxx_hidden_ProcessingLevel     v1.ProcessingLevel     `protobuf:"varint,2,opt,name=processing_level,json=processingLevel,enum=datasets.v1.ProcessingLevel"`
  // more fields
}
```

Notice that the fields are private (starting with a lowercase letter), so they are not accessible.
Protobuf hides the fields and provides getters and setters to access them.

## Protobuf 101

### Initializing a message

Here is how to initialize a `v1.Sentinel1Sar` message.

```go Go theme={"system"}
import (
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"
)

datapoint := v1.Sentinel1Sar_builder{
	Time:        timestamppb.New(time.Now()),
	GranuleName: proto.String("S1A_EW_GRDH_1SSH_20141004T020507_20141004T020611_002673_002FAF_8645_COG.SAFE"),
	ProductType: proto.String("EW_GRDH_1S-COG"),
	FileSize:    proto.Int64(488383473),
}.Build()
```

Protobuf fields are private and provides a builder pattern to create a message.

`proto.String` is a helper function that converts `string` to `*string`.
This allows protobuf to differentiate between a field that is set to an empty string and a field that is not set (nil).
An exhaustive list of those helper functions can be found [here](https://github.com/golang/protobuf/blob/master/proto/wrappers.go).

Only primitives have a `proto.XXX` helper function.
Complex types such as timestamps, durations, UUIDs, and geometries have a [constructor function](#constructors).

### Getters and setters

Protobuf provides methods to get, set, clear and check if a field is set.

```go Go theme={"system"}
fmt.Println(datapoint.GetGranuleName())

datapoint.SetGranuleName("my amazing granule")

datapoint.ClearGranuleName()

if datapoint.HasGranuleName() {
  fmt.Println("Granule name is set")
}
```

Getters for primitive types will return a Go native type (for example, int64, string, etc.).
Getters for complex types such as timestamps, durations, UUIDs, and geometries can also be converted to more standard types using [AsXXX](#asxxx-methods) methods.

## Well known types

Beside Go primitives, Tilebox supports some well known types:

* Duration: A duration of time. See [Duration](https://protobuf.dev/reference/protobuf/google.protobuf/#duration) for more information.
* Timestamp: A point in time. See [Timestamp](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp) for more information.
* UUID: A [universally unique identifier (UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier).
* Geometry: Geospatial geometries of type Point, LineString, Polygon or MultiPolygon.

They have a couple of useful methods to work with them.

### Constructors

```go Go theme={"system"}
import (
	"github.com/paulmach/orb"
	datasetsv1 "github.com/tilebox/tilebox-go/protogen/datasets/v1"
	"google.golang.org/protobuf/types/known/durationpb"
	"google.golang.org/protobuf/types/known/timestamppb"
)

timestamppb.New(time.Now())
durationpb.New(10 * time.Second)
datasetsv1.NewUUID(uuid.New())
datasetsv1.NewGeometry(orb.Point{1, 2})
```

### `CheckValid` method

`CheckValid` returns an error if the field is invalid.

```go Go theme={"system"}
err := datapoint.GetTime().CheckValid()
if err != nil {
  fmt.Println(err)
}
```

### `IsValid` method

`IsValid` reports whether the field is valid. It's equivalent to `CheckValid == nil`.

```go Go theme={"system"}
if datapoint.GetTime().IsValid() {
  fmt.Println("Valid")
}
```

### `AsXXX` methods

`AsXXX` methods convert the field to a more user friendly type.

* `AsUUID` will convert a `datasetsv1.UUID` field to a [uuid.UUID](https://pkg.go.dev/github.com/google/uuid#UUID) type
* `AsTime` will convert a `timestamppb.Timestamp` field to a [time.Time](https://pkg.go.dev/time#Time) type
* `AsDuration` will convert a `durationpb.Duration` field to a [time.Duration](https://pkg.go.dev/time#Duration) type
* `AsGeometry` will convert a `datasetsv1.Geometry` field to an [orb.Geometry](https://github.com/paulmach/orb?tab=readme-ov-file#shared-geometry-interface) interface

```go Go theme={"system"}
datapoint.GetId().AsUUID() // uuid.UUID
datapoint.GetTime().AsTime() // time.Time
datapoint.GetDuration().AsDuration() // time.Duration
datapoint.GetGeometry().AsGeometry() // orb.Geometry
```

Those methods performs conversion on a best-effort basis. Type validity must be checked beforehand using `IsValid` or `CheckValid` methods.

## Common data operations

Datapoints are contained in a standard Go slice so all the usual [slice operations](https://gobyexample.com/slices) and [slice functions](https://pkg.go.dev/slices) can be used.

The usual pattern to iterate over data in Go is by using a `for` loop.

As an example, here is how to extract the `copernicus_id` fields from the datapoints.

```go Go theme={"system"}
// assuming datapoints has been filled using `client.Datapoints.QueryInto` method
var datapoints []*v1.Sentinel1Sar

copernicusIDs := make([]uuid.UUID, len(datapoints))
for i, dp := range datapoints {
  copernicusIDs[i] = dp.GetCopernicusId().AsUUID()
}
```

Here is an example of filtering out datapoints that have been published before January 2000 and are not from the Sentinel-1C platform.

```go Go theme={"system"}
jan2000 := time.Date(2000, time.January, 1, 0, 0, 0, 0, time.UTC)
// slice of length of 0, but preallocate a capacity of len(datapoints)
s1cDatapoints := make([]*v1.Sentinel1Sar, 0, len(datapoints))

for _, dp := range datapoints {
  if dp.GetPublished().AsTime().Before(jan2000) {
    continue
  }
  if dp.GetPlatform() != "S1C" {
    continue
  }

  s1cDatapoints = append(s1cDatapoints, proto.CloneOf(dp)) // Copy the underlying data
}
```

## Converting to JSON

Protobuf messages can be converted to JSON without loss of information. This is useful for interoperability with other systems that doesn't use protobuf.
A guide on protoJSON can be found format here: [https://protobuf.dev/programming-guides/json/](https://protobuf.dev/programming-guides/json/)

```go Go theme={"system"}
originalDatapoint := datapoints[0]

// Convert proto.Message to JSON as bytes
jsonDatapoint, err := protojson.Marshal(originalDatapoint)
if err != nil {
  log.Fatalf("Failed to marshal datapoint: %v", err)
}
fmt.Println(string(jsonDatapoint))
```

```plaintext Output theme={"system"}
{"time":"2001-01-01T00:00:00Z","id":{"uuid":"AOPHpzQAAmV2MZ4+Zv+JGg=="},"ingestionTime":"2025-03-25T10:26:10.577385176Z","granuleName":"MCD12Q1.A2001001.h02v08.061.2022146033342.hdf","geometry":{"wkb":"AQMAAAABAAAABQAAAFIi9vf7TmTAXsX3////I0Bexff///9jwAAAAAAAAAAACUn4//+/YsAAAAAAAAAAAC7AdjgMCmPAXsX3////I0BSIvb3+05kwF7F9////yNA"},"endTime":"2001-12-31T23:59:59Z","horizontalTileNumber":"2","verticalTileNumber":"8","tileId":"51002008","fileSize":"176215","checksum":"771212892","checksumType":"CKSUM","dayNightFlag":"Day","publishedAt":"2022-06-23T10:58:13.895Z"}
```

It can also be converted back to a `proto.Message`.

```go Go theme={"system"}
// Convert JSON bytes to proto.Message
unmarshalledDatapoint := &v1.Sentinel1Sar{}
err = protojson.Unmarshal(jsonDatapoint, unmarshalledDatapoint)
if err != nil {
  log.Fatalf("Failed to unmarshal datapoint: %v", err)
}

fmt.Println("Are both equal?", proto.Equal(unmarshalledDatapoint, originalDatapoint))
```

```plaintext Output theme={"system"}
Are both equal? true
```


# Tilebox languages and SDKs
Source: https://docs.tilebox.com/sdks/introduction

Official SDKs for Python and Go that provide full access to Tilebox datasets, storage, and workflow orchestration in your preferred programming language.

Use Tilebox SDKs when you want Tilebox in notebooks, scripts, services, or workflow task code. The SDKs expose datasets, storage access, workflow jobs, runners, observability queries, and automation APIs from regular app code.

If you are setting up a coding agent, start with [Onboard your agent](/onboard-your-agent). Agents use the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) first, then inspect SDK examples or API reference pages when they need to edit code.

<Columns>
  <HeroCard title="Tilebox Python" description="The Tilebox Python SDK includes full support for datasets, storage and workflows. Use it together with your favorite Python libraries to access and process data." href="/sdks/python/install">
    <img alt="Tilebox Python" />
  </HeroCard>

  <HeroCard title="Tilebox Go" description="Take advantage of strongly typed datasets and workflows - including your own, custom dataset types - by using the Tilebox Go SDK." href="/sdks/go/install">
    <img alt="Tilebox Go" />
  </HeroCard>
</Columns>

## Choose an SDK

<Columns>
  <Card title="Python notebooks and data workflows" icon="python" href="/sdks/python/install">
    Use Python for notebooks, xarray-based dataset queries, data ingestion, and Python workflow task code.
  </Card>

  <Card title="Go services and typed integrations" icon="golang" href="/sdks/go/install">
    Use Go for backend services, typed dataset access, workflow task submission, and long-running runners.
  </Card>

  <Card title="Sample notebooks" icon="book" href="/sdks/python/sample-notebooks">
    Start from working Python notebooks for common Tilebox dataset workflows.
  </Card>

  <Card title="API reference" icon="code" href="/api-reference/python/tilebox.datasets/Client">
    Look up method signatures and parameter details for Python and Go clients.
  </Card>
</Columns>


# Async support
Source: https://docs.tilebox.com/sdks/python/async

The Tilebox Python SDK offers full async support with asyncio, allowing you to run data operations in parallel for significantly better performance.

## Why use async?

When working with external datasets, such as [Tilebox datasets](/datasets/concepts/datasets), loading data may take some time. To speed up this process, you can run requests in parallel. While you can use multi-threading or multi-processing, which can be complex, often times a simpler option is to perform data loading tasks asynchronously using coroutines and `asyncio`.

## Switching to an async datasets client

To switch to the async client, change the import statement for the `Client`. The example below illustrates this change.

<CodeGroup>
  ```python Python (Sync) theme={"system"}
  from tilebox.datasets import Client

  # This client is synchronous
  client = Client()
  ```

  ```python Python (Async) theme={"system"}
  from tilebox.datasets.aio import Client

  # This client is asynchronous
  client = Client()
  ```
</CodeGroup>

After switching to the async client, use `await` for operations that interact with the Tilebox API.

<CodeGroup>
  ```python Python (Sync) theme={"system"}
  # Listing datasets
  datasets = client.datasets()

  # Listing collections
  dataset = datasets.open_data.copernicus.sentinel1_sar
  collections = dataset.collections()

  # Collection information
  collection = collections["S1A_IW_RAW__0S"]
  info = collection.info()
  print(f"Data for My-collection is available for {info.availability}")

  # Loading data
  data = collection.query(temporal_extent=("2022-05-01", "2022-06-01"), show_progress=True)

  # Finding a specific datapoint
  datapoint_uuid = "01910b3c-8552-7671-3345-b902cc0813f3"
  datapoint = collection.find(datapoint_uuid)
  ```

  ```python Python (Async) theme={"system"}
  # Listing datasets
  datasets = await client.datasets()

  # Listing collections
  dataset = datasets.open_data.copernicus.sentinel1_sar
  collections = await dataset.collections()

  # Collection information
  collection = collections["S1A_IW_RAW__0S"]
  info = await collection.info()
  print(f"Data for My-collection is available for {info.availability}")

  # Loading data
  data = await collection.query(temporal_extent=("2022-05-01", "2022-06-01"), show_progress=True)

  # Finding a specific datapoint
  datapoint_uuid = "01910b3c-8552-7671-3345-b902cc0813f3"
  datapoint = await collection.find(datapoint_uuid)
  ```
</CodeGroup>

<Note>
  Jupyter notebooks and similar interactive environments support asynchronous code execution. You can use
  `await some_async_call()` as the output of a code cell.
</Note>

## Fetching data concurrently

The primary benefit of the async client is that it allows concurrent requests, enhancing performance.
In below example, data is fetched from multiple collections. The synchronous approach retrieves data sequentially, while the async approach does so concurrently, resulting in faster execution.

<CodeGroup>
  ```python Python (Sync) theme={"system"}
  # Example: fetching data sequentially

  # switch to the async example to compare the differences
  import time
  from tilebox.datasets import Client
  from tilebox.datasets.sync.timeseries import TimeseriesCollection

  client = Client()
  datasets = client.datasets()
  collections = datasets.open_data.copernicus.landsat8_oli_tirs.collections()

  def stats_for_2020(collection: TimeseriesCollection) -> None:
      """Fetch data for 2020 and print the number of data points that were loaded."""
      data = collection.query(temporal_extent=("2020-01-01", "2021-01-01"), show_progress=True)
      n = data.sizes['time'] if 'time' in data else 0
      return (collection.name, n)

  start = time.monotonic()
  results = [stats_for_2020(collections[name]) for name in collections]
  duration = time.monotonic() - start

  for collection_name, n in results:
      print(f"There are {n} datapoints in {collection_name} for 2020.")
  print(f"Fetching data took {duration:.2f} seconds")
  ```

  ```python Python (Async) theme={"system"}
  # Example: fetching data concurrently

  import asyncio
  import time
  from tilebox.datasets.aio import Client
  from tilebox.datasets.aio.timeseries import TimeseriesCollection

  client = Client()
  datasets = await client.datasets()
  collections = await datasets.open_data.copernicus.landsat8_oli_tirs.collections()

  async def stats_for_2020(collection: TimeseriesCollection) -> None:
      """Fetch data for 2020 and print the number of data points that were loaded."""
      data = await collection.query(temporal_extent=("2020-01-01", "2021-01-01"), show_progress=True)
      n = data.sizes['time'] if 'time' in data else 0
      return (collection.name, n)

  start = time.monotonic()

  # Initiate all requests concurrently
  requests = [stats_for_2020(collections[name]) for name in collections]
  # Wait for all requests to finish in parallel
  results = await asyncio.gather(*requests)

  duration = time.monotonic() - start

  for collection_name, n in results:
      print(f"There are {n} datapoints in {collection_name} for 2020.")
  print(f"Fetching data took {duration:.2f} seconds")
  ```
</CodeGroup>

The output demonstrates that the async approach runs approximately 30% faster for this example. With `show_progress` enabled, the progress bars update concurrently.

<CodeGroup>
  ```plaintext Python (Sync) theme={"system"}
  There are 19624 datapoints in L1GT for 2020.
  There are 1281 datapoints in L1T for 2020.
  There are 65313 datapoints in L1TP for 2020.
  There are 25375 datapoints in L2SP for 2020.
  Fetching data took 10.92 seconds
  ```

  ```plaintext Python (Async) theme={"system"}
  There are 19624 datapoints in L1GT for 2020.
  There are 1281 datapoints in L1T for 2020.
  There are 65313 datapoints in L1TP for 2020.
  There are 25375 datapoints in L2SP for 2020.
  Fetching data took 7.45 seconds
  ```
</CodeGroup>

## Async workflows

The Tilebox workflows Python client does not have an async client. This is because workflows are designed for distributed and concurrent execution outside a single async event loop. But within a single task, you may use still use`async` code to take advantage of asynchronous execution, such as parallel data loading. You can achieve this by wrapping your async code in `asyncio.run`.

Below is an example of using async code within a workflow task.

<CodeGroup>
  ```python Python (Async) theme={"system"}
  import asyncio
  import xarray as xr

  from tilebox.datasets.aio import Client as DatasetsClient
  from tilebox.datasets.query import TimeIntervalLike
  from tilebox.workflows import Task, ExecutionContext

  class FetchData(Task):
      def execute(self, context: ExecutionContext) -> None:
          # The task execution itself is synchronous
          # But we can leverage async code within the task using asyncio.run

          # This will fetch three months of data in parallel
          data_jan, data_feb, data_mar = asyncio.run(load_first_three_months())
          
  async def load_data(interval: TimeIntervalLike):
      datasets = await DatasetsClient().datasets()
      collections = await datasets.open_data.copernicus.landsat8_oli_tirs.collections()
      return await collections["L1T"].query(temporal_extent=interval)

  async def load_first_three_months() -> tuple[xr.Dataset, xr.Dataset, xr.Dataset]:
      jan = load_data(("2020-01-01", "2020-02-01"))
      feb = load_data(("2020-02-01", "2020-03-01"))
      mar = load_data(("2020-03-01", "2020-04-01"))
      # load the three months in parallel
      jan, feb, mar = await asyncio.gather(jan, feb, mar)
      return jan, feb, mar
  ```
</CodeGroup>

<Tip>
  If you encounter an error like `RuntimeError: asyncio.run() cannot be called from a running event loop`, it means you're trying to start another asyncio event loop (with `asyncio.run`) from within an existing one. This often happens in Jupyter notebooks since they automatically start an event loop. A way to resolve this is by using [nest-asyncio](https://pypi.org/project/nest-asyncio/).
</Tip>


# Installation
Source: https://docs.tilebox.com/sdks/python/install

Install the Tilebox Python SDK packages using your preferred package manager to start working with datasets, workflows, and satellite data storage.

## Package Overview

Tilebox offers a Python SDK for accessing Tilebox services. The SDK includes separate packages that can be installed individually based on the services you wish to use, or all together for a comprehensive experience.

<Columns>
  <Card title="tilebox-datasets" icon="database" href="https://pypi.org/project/tilebox-datasets/">
    Access Tilebox datasets from Python
  </Card>

  <Card title="tilebox-workflows" icon="network-wired" href="https://pypi.org/project/tilebox-workflows/">
    Workflow client and runner for Tilebox
  </Card>
</Columns>

## Installation

Install the Tilebox python packages using your preferred package manager.

<Tip>
  For new projects Tilebox recommend using [uv](https://docs.astral.sh/uv/).
</Tip>

<CodeGroup>
  ```bash uv theme={"system"}
  uv add tilebox-datasets tilebox-workflows tilebox-storage
  ```

  ```bash pip theme={"system"}
  pip install tilebox-datasets tilebox-workflows tilebox-storage
  ```

  ```bash poetry theme={"system"}
  poetry add tilebox-datasets="*" tilebox-workflows="*" tilebox-storage="*"
  ```

  ```bash pipenv theme={"system"}
  pipenv install tilebox-datasets tilebox-workflows tilebox-storage
  ```
</CodeGroup>

## Setting up a local JupyterLab environment

<Tip>
  To get started quickly, you can also use an existing Jupyter-compatible cloud environment such as [Google Colab](https://colab.research.google.com/).
</Tip>

If you want to set up a local Jupyter environment to explore the SDK or to run the [Sample notebooks](/sdks/python/sample-notebooks) locally, install [JupyterLab](https://github.com/jupyterlab/jupyterlab) for a browser-based development environment. It's advised to install the Tilebox packages along with JupyterLab, [ipywidgets](https://ipywidgets.readthedocs.io/en/latest/user_install.html), and [tqdm](https://tqdm.github.io/) for an enhanced experience.

```uv uv theme={"system"}
mkdir tilebox-exploration
cd tilebox-exploration

uv init --no-package
uv add tilebox-datasets tilebox-workflows tilebox-storage
uv add jupyterlab ipywidgets tqdm
uv run jupyter lab
```

### Trying it out

After installation, create a new notebook and paste the following code snippet to verify your installation. If you're new to Jupyter, you can refer to the guide on [interactive environments](/sdks/python/sample-notebooks#interactive-environments).

```python Python theme={"system"}
from tilebox.datasets import Client

client = Client()
datasets = client.datasets()

collection = datasets.open_data.copernicus.landsat8_oli_tirs.collection("L1T")
data = collection.query(temporal_extent=("2015-01-01", "2020-01-01"), show_progress=True)
data
```

If the installation is successful, the output should appear as follows.

<Frame>
  <img alt="Local JupyterLab" />

  <img alt="Local JupyterLab" />
</Frame>


# Sample notebooks
Source: https://docs.tilebox.com/sdks/python/sample-notebooks

Get started quickly with ready-to-run Jupyter notebooks that demonstrate common satellite data access and analysis workflows using the Tilebox Python SDK.

To quickly become familiar with the Python client, you can explore some sample notebooks. Each notebook can be executed standalone from top to bottom.

## Sample notebooks

You can access the sample notebooks on [<Icon icon="google-drive" /> Google Drive](https://drive.google.com/drive/folders/1I7G35LLeB2SmKQJFsyhpSVNyZK9uOeJt).

<Tip>
  Right click a notebook in Google Drive and select `Open with -> Google Colaboratory` to open it directly in the browser using [Google Colab](https://colab.research.google.com/).
</Tip>

More examples can be found throughout the docs.

### Notebook overview

<AccordionGroup>
  <Accordion title="ERS Opendata Access">
    This notebook demonstrates how to use to query metadata from the ERS-SAR Opendata dataset. It shows how to filter results by geographical location and download product data for a specific granule.

    [<Icon icon="arrow-up-right-from-square" /> Open in
    Colab](https://colab.research.google.com/drive/1LTYhLKy8m9psMhu0DvANs7hyS3ViVpas)
  </Accordion>

  <Accordion title="S5P Tropomi Methane Data Access">
    This notebook illustrates how to query the S5P Tropomi Opendata dataset for methane products. It also explains how to filter results based on geographical location and download product data for a specific granule.

    [<Icon icon="arrow-up-right-from-square" /> Open in
    Colab](https://colab.research.google.com/drive/1eVYARNFnTIeQqBs6gqeay01EDvRk2EI4)
  </Accordion>

  <Accordion title="MODIS-based Dataset Ingestion">
    This notebook demonstrates how to ingest data into a Dataset. In this case it's using a prepared sample dataset from the [MODIS instrument](https://lpdaac.usgs.gov/products/mcd12q1v006/).

    [<Icon icon="arrow-up-right-from-square" /> Open in
    Colab](https://colab.research.google.com/drive/1QS-srlWPMJg4csc0ycn36yCX9U6mvIpW)
  </Accordion>

  <Accordion title="Sentinel-2 Cloud-free Mosaic">
    Created with Tilebox Workflows, this 10m resolution mosaic highlights distributed, auto-parallelizing capabilities.
    Data from `Copernicus Dataspace` was reprojected on `CloudFerro` (intermediate products on AWS S3), and the final composite was built locally using auto-parallelized team notebooks.

    [<Icon icon="arrow-up-right-from-square" /> Open the Mosaic](https://examples.tilebox.com/sentinel2_mosaic)

    [<Icon icon="arrow-up-right-from-square" /> Open in
    Github](https://github.com/tilebox/examples/tree/main/s2-cloudfree-mosaic)
  </Accordion>

  <Accordion title="Generate VCI and FPAR visualizations">
    A workflow for calculating the Vegetation Condition Index (VCI) from FPAR data. It can be run on one or more local machines or on a cloud cluster.

    [<Icon icon="arrow-up-right-from-square" /> View on YouTube](https://youtu.be/s4wzyX9adWo)

    [<Icon icon="arrow-up-right-from-square" /> Open in
    Github](https://github.com/tilebox/fpar-based-vci-example)
  </Accordion>
</AccordionGroup>

Execute cells one by one using `Shift+Enter`. Most commonly used libraries are pre-installed.

<Note>All demo notebooks require Python 3.10 or higher.</Note>

## Interactive environments

Jupyter, Google Colab, and JetBrains Datalore are interactive environments that simplify the development and sharing of algorithmic code. They allow users to work with notebooks, which combine code and rich text elements like figures, links, and equations. Notebooks require no setup and can be easily shared.

<AccordionGroup>
  <Accordion title="Jupyter">
    [Jupyter notebooks](https://jupyter.org/) are the original interactive environment for Python. They are useful but require local installation.
  </Accordion>

  <Accordion title="Google Colab">
    [Google Colab](https://colab.research.google.com/) is a free tool that provides a hosted interactive Python environment. It easily connects to local Jupyter instances and allows code sharing using Google credentials or within organizations using Google Workspace.
  </Accordion>

  <Accordion title="JetBrains Datalore">
    [JetBrains Datalore](https://datalore.jetbrains.com/) is a free platform for collaborative testing, development, and sharing of Python code and algorithms. It has built-in secret management for storing credentials. Datalore also features advanced JetBrains syntax highlighting and autocompletion. Currently, it only supports Python 3.8, which is not compatible with the Tilebox Python client.
  </Accordion>
</AccordionGroup>

Since Colab is a hosted free tool that meets all requirements, including Python ≥3.10, it's recommended for use.

## Installing packages

Within your interactive environment, you can install missing packages using pip in "magic" cells, which start with an exclamation mark.

```bash theme={"system"}
# pip is already installed in your interactive environment
!pip3 install ....
```

All APIs or commands that require authentication can be accessed through client libraries that hide tokens, allowing notebooks to be shared without exposing personal credentials.

## Executing code

Execute code by clicking the play button in the top left corner of the cell or by pressing `Shift + Enter`. While the code is running, a spinning icon appears. When the execution finishes, the icon changes to a number, indicating the order of execution. The output displays below the code.

## Authorization

When sharing notebooks, avoid directly sharing your Tilebox API key. Instead, use one of two methods to authenticate the Tilebox Python client in interactive environments: through environment variables or interactively.

```python Using environment variables to store your API key theme={"system"}
# Define an environment variable "TILEBOX_API_KEY" that contains your API key
import os
token = os.getenv("TILEBOX_API_KEY")
```

**Interactive** authorization is possible using the built-in `getpass` module. This prompts the user for the API key when running the code, storing it in memory without sharing it when the notebook is shared.

```python Interactively providing your API key theme={"system"}
from getpass import getpass
token = getpass("API key:")
```


# Xarray
Source: https://docs.tilebox.com/sdks/python/xarray

Tilebox uses Xarray as its primary data structure for representing and analyzing multi-dimensional satellite data with labeled dimensions and coordinates.

[example_satellite_data.nc]: https://github.com/tilebox/docs/raw/main/assets/data/example_satellite_data.nc

[Xarray](https://xarray.dev/) is a library designed for working with labeled multi-dimensional arrays. Built on top of [NumPy](https://numpy.org/) and [Pandas](https://pandas.pydata.org/), Xarray adds labels in the form of dimensions, coordinates, and attributes, enhancing the usability of raw NumPy-like arrays. This enables a more intuitive, concise, and less error-prone development experience. The library also includes a large and expanding collection of functions for advanced analytics and visualization.

<Frame>
  <img alt="Overview of the Xarray data structure" />
</Frame>

<Note>
  An overview of the Xarray library and its suitability for N-dimensional data (such as Tilebox time series datasets) is available in the official [Why Xarray? documentation page](https://xarray.pydata.org/en/stable/why-xarray.html).
</Note>

The Tilebox Python client provides access to satellite data as an [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html#xarray.Dataset). This approach offers a great number of benefits over custom Tilebox-specific data structures:

<AccordionGroup>
  <Accordion title="Familiarity">
    Xarray is based on NumPy and Pandas—two of the most widely used Python libraries for scientific computing. Familiarity with these libraries translates well to using Xarray.
  </Accordion>

  <Accordion title="Performance">
    Leveraging NumPy, which is built on C and Fortran, Xarray benefits from extensive performance optimizations. This allows Xarray to efficiently handle large datasets.
  </Accordion>

  <Accordion title="Interoperability">
    As a widely used library, Xarray easily integrates with many other libraries. Many third-party libraries are also available to expand Xarray's capabilities for diverse use cases.
  </Accordion>

  <Accordion title="Flexibility">
    Xarray is versatile and supports a broad range of applications. It's also easy to extend with custom features.
  </Accordion>
</AccordionGroup>

## Example dataset

To understand how Xarray functions, below is a quick a look at a sample dataset as it might be retrieved from a [Tilebox datasets](/datasets/concepts/datasets) client.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.datasets import Client

  client = Client()
  datasets = client.datasets()
  collection = datasets.open_data.copernicus.landsat8_oli_tirs.collection("L1GT")
  satellite_data = collection.query(temporal_extent=("2022-05-01", "2022-06-01"), show_progress=True)
  print(satellite_data)
  ```
</CodeGroup>

```plaintext Output theme={"system"}
<xarray.Dataset> Size: 305kB
Dimensions:                (time: 514, latlon: 2)
Coordinates:
    ingestion_time         (time) datetime64[ns] 4kB 2024-07-22T09:06:43.5586...
    id                     (time) <U36 74kB '01807eaa-86f8-2a72-1a03-794e7a55...
  * time                   (time) datetime64[ns] 4kB 2022-05-01T08:09:06.5520...
  * latlon                 (latlon) <U9 72B 'latitude' 'longitude'
Data variables: (12/28)
    granule_name           (time) object 4kB 'LC08_L1GT_175018_20220501_20220...
    processing_level       (time) <U2 4kB 'L1' 'L1' 'L1' 'L1' ... 'L1' 'L1' 'L1'
    satellite              (time) object 4kB 'LANDSAT-8' ... 'LANDSAT-8'
    product_type           (time) object 4kB 'L1GT' 'L1GT' ... 'L1GT' 'L1GT'
    copernicus_id          (time) <U36 74kB '2181f9f6-1ef0-510c-b715-0f299320...
    online                 (time) bool 514B True True True ... True True True
    ...                     ...
    resolution             (time) int64 4kB 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0
    bands                  (time) int64 4kB 12 12 12 12 12 12 ... 12 12 12 12 12
    path                   (time) int64 4kB 175 175 175 175 ... 209 209 209 209
    row                    (time) int64 4kB 18 30 31 32 27 32 ... 18 21 22 23 24
    sun_azimuth            (time) float64 4kB 164.0 144.9 143.0 ... 153.5 151.6
    sun_elevation          (time) float64 4kB 44.2 57.78 58.76 ... 56.72 57.82
```

<Note>
  This basic dataset illustrates common use cases for Xarray. To follow along, you can [download the dataset as a NetCDF file][example_satellite_data.nc]. The [Reading and writing files section](/sdks/python/xarray#reading-and-writing-files) explains how to save and load Xarray datasets from NetCDF files.
</Note>

Here's an overview of the output:

* The `satellite_data` **dataset** contains **dimensions**, **coordinates**, and **variables**.
* The `time` **dimension** has 514 elements, indicating that there are 514 data points in the dataset.
* The `time` **dimension coordinate** contains datetime values representing when the data was measured. The `*` indicates a dimension coordinate, which enables label-based indexing and alignment.
* The `ingestion_time` **non-dimension coordinate** holds datetime values for when the data was ingested into Tilebox. Non-dimension coordinates carry coordinate data but are not used for label-based indexing. They can even be [multidimensional](https://docs.xarray.dev/en/stable/examples/multidimensional-coords.html).
* The dataset includes 28 **variables**.
* The `bands` **variable** contains integers indicating how many bands the data contains.
* The `sun_elevation` **variable** contains floating-point values representing the sun's elevation when the data was measured.

<Note>
  Explore the [xarray terminology overview](https://docs.xarray.dev/en/stable/user-guide/terminology.html) to broaden your understanding of **datasets**, **dimensions**, **coordinates**, and **variables**.
</Note>

## Accessing data in a dataset

### By index

You can access data in different ways. The Xarray documentation offers a [comprehensive overview](https://docs.xarray.dev/en/stable/user-guide/indexing.html) of these methods.

To access the `sun_elevation` variable:

```python Accessing values theme={"system"}
# Print the first sun elevation value
print(satellite_data.sun_elevation[0])
```

```plaintext Output theme={"system"}
<xarray.DataArray 'sun_elevation' ()> Size: 8B
array(44.19904463)
Coordinates:
    ingestion_time   datetime64[ns] 8B 2024-07-22T09:06:43.558629
    id               <U36 144B '01807eaa-86f8-2a72-1a03-794e7a556271'
    time             datetime64[ns] 8B 2022-05-01T08:09:06.552000
```

In the output, the first sun elevation value is `44.19904463`. It appears as an `xarray.DataArray` object to allow access to the corresponding coordinates. To retrieve the plain Python object, use the `item()` method:

```python Accessing raw values theme={"system"}
sun_elevation = satellite_data.sun_elevation[0].item()
print(sun_elevation)
```

```python Output theme={"system"}
44.19904463
```

You can access coordinates similarly. For datetime fields, Xarray provides a special `dt` (datetime) accessor for formatting time as a string:

```python Accessing and formatting datetime fields theme={"system"}
time_format = "%Y-%m-%d %H:%M:%S"
time = satellite_data.time[0].dt.strftime(time_format).item()
ingestion_time = satellite_data.ingestion_time[0].dt.strftime(time_format).item()
print(f"Measurement 0 was taken at {time} and ingested at {ingestion_time}")
```

```plaintext Output theme={"system"}
Measurement 0 was taken at 2022-05-01 08:09:06 and ingested at 2024-07-22 09:06:43
```

You can also retrieve an entire dataset containing all variables and coordinates for a single data point using the `isel` method (index selection):

```python Accessing a whole datapoint by index theme={"system"}
datapoint = satellite_data.isel(time=0)
print(datapoint)
```

```plaintext Output theme={"system"}
<xarray.Dataset> Size: 665B
Dimensions:                (latlon: 2)
Coordinates:
    ingestion_time         datetime64[ns] 8B 2024-07-22T09:06:43.558629
    id                     <U36 144B '01807eaa-86f8-2a72-1a03-794e7a556271'
    time                   datetime64[ns] 8B 2022-05-01T08:09:06.552000
  * latlon                 (latlon) <U9 72B 'latitude' 'longitude'
Data variables: (12/28)
    granule_name           object 8B 'LC08_L1GT_175018_20220501_20220504_02_T2'
    processing_level       <U2 8B 'L1'
    satellite              object 8B 'LANDSAT-8'
    ...                     ...
```

### Subsets of data

You can access subsets of the data as well. Here are methods to retrieve the first three and last three sun elevations.

```python Accessing raw values theme={"system"}
# Individual variables
first_3_sun_elevations = satellite_data.sun_elevation[0:3]
print("First 3 sun elevations", first_3_sun_elevations.values)
last_3_sun_elevations = satellite_data.sun_elevation[-3:]
print("Last 3 sun elevations", last_3_sun_elevations.values)

# Whole sub datasets
first_3 = satellite_data.isel(time=slice(0, 3))
last_3 = satellite_data.isel(time=slice(-3, None))
print("Sub dataset of the last 3 data points")
print(last_3)
```

```plaintext Output theme={"system"}
First 3 sun elevations [44.19904463 57.77561083 58.76316786]
Last 3 sun elevations [55.60690523 56.72453179 57.81917624]

Sub dataset of the last 3 data points
<xarray.Dataset> Size: 2kB
Dimensions:                (time: 3, latlon: 2)
Coordinates:
    ingestion_time         (time) datetime64[ns] 24B 2024-07-22T09:08:24.7395...
    id                     (time) <U36 432B '018119eb-5291-edbc-381e-ce71e885...
  * time                   (time) datetime64[ns] 24B 2022-05-31T11:41:01.4570...
  * latlon                 (latlon) <U9 72B 'latitude' 'longitude'
Data variables: (12/28)
    granule_name           (time) object 24B 'LC08_L1GT_209022_20220531_20220...
    processing_level       (time) <U2 24B 'L1' 'L1' 'L1'
    satellite              (time) object 24B 'LANDSAT-8' 'LANDSAT-8' 'LANDSAT-8'
    ...                     ...
```

## Filtering data

Xarray allows convenient filtering of datasets based on conditions. For example, filter a dataset to only include sun elevation values where cloud cover is `0`:

```python Filtering data by sensor theme={"system"}
without_cloud = satellite_data.sun_elevation[satellite_data.cloud_cover == 0]
print(without_cloud)
```

```plaintext Output theme={"system"}
<xarray.DataArray 'sun_elevation' (time: 27)> Size: 216B
array([63.89629314, 63.35038654, ..., 64.37400345, 64.37400345])
Coordinates:
    ingestion_time  (time) datetime64[ns] 216B 2024-07-22T09:06:43.558629 ......
    id              (time) <U36 4kB '01807f66-8411-2e5d-719b-ce51152175eb' .....
  * time            (time) datetime64[ns] 216B 2022-05-01T11:34:26.577000 ......
```

You can combine conditions to filter for sun elevation values between `45` and `90` with cloud cover `0`:

```python Filtering data by cloud_cover and sun_elevation value theme={"system"}
data_filter = (
    (satellite_data.cloud_cover == 0) &
    (satellite_data.sun_elevation > 45) &
    (satellite_data.sun_elevation < 90)
)
filtered_sun_elevations = satellite_data.sun_elevation[data_filter]
print(filtered_sun_elevations)
```

```plaintext Output theme={"system"}
<xarray.DataArray 'sun_elevation' (time: 27)> Size: 216B
array([63.89629314, 63.35038654, ..., 64.37400345, 64.37400345])
Coordinates:
    ingestion_time  (time) datetime64[ns] 216B 2024-07-22T09:06:43.558629 ......
    id              (time) <U36 4kB '01807f66-8411-2e5d-719b-ce51152175eb' .....
  * time            (time) datetime64[ns] 216B 2022-05-01T11:34:26.577000 ......
```

## Selecting data by value

You can use dimension coordinate values to index your dataset. For instance, access the data point recorded at `2022-05-01T11:28:28.249000`:

<Tip>
  Selecting data by value requires unique coordinate values. In case of duplicates, you will encounter an `InvalidIndexError`. To avoid this, you can [drop duplicates](#dropping-duplicates).
</Tip>

```python Indexing by time theme={"system"}
specific_datapoint = satellite_data.sel(time="2022-05-01T11:28:28.249000")
print(specific_datapoint)
```

```plaintext Output theme={"system"}
<xarray.Dataset> Size: 665B
Dimensions:                (latlon: 2)
Coordinates:
    ingestion_time         datetime64[ns] 8B 2024-07-22T09:06:43.558629
    id                     <U36 144B '01807f61-0c59-99ed-8e33-c5d8ed6e7879'
    time                   datetime64[ns] 8B 2022-05-01T11:28:28.249000
  * latlon                 (latlon) <U9 72B 'latitude' 'longitude'
Data variables: (12/28)
    granule_name           object 8B 'LC08_L1GT_207022_20220501_20220501_02_T2'
    processing_level       <U2 8B 'L1'
    ...                     ...
```

Attempting to access a value not in the dataset raises a `KeyError`.

```python Indexing by time (not found) theme={"system"}
nearest_datapoint = satellite_data.sel(time="2022-05-01T11:28:28.000000")
>>> raises KeyError: "2022-05-01T11:28:28.000000"
```

To return the closest value instead of raising an error, specify a `method`.

```python Finding the closest data point theme={"system"}
nearest_datapoint = satellite_data.sel(time="2022-05-01T11:28:28.000000", method="nearest")
assert nearest_datapoint.equals(specific_datapoint)  # passes
```

## Dropping duplicates

Xarray allows you to drop duplicate values from a dataset. For example, to drop duplicate timestamps:

```python Dropping duplicates theme={"system"}
deduped = satellite_data.drop_duplicates("time")
```

De-duped datasets are required for certain operations, like [selecting data by value](#selecting-data-by-value).

## Statistics

Xarray and NumPy include a wide range of statistical functions that you can apply to a dataset or DataArray. Here are some examples:

```python Computing dataset statistics theme={"system"}
cloud_cover = satellite_data.cloud_cover
min_meas = cloud_cover.min().item()
max_meas = cloud_cover.max().item()
mean_meas = cloud_cover.mean().item()
std_meas = cloud_cover.std().item()
print(f"Cloud cover ranges from {min_meas:.2f} to {max_meas:.2f} with a mean of {mean_meas:.2f} and a standard deviation of {std_meas:.2f}")
```

```plaintext Output theme={"system"}
Cloud cover ranges from 0.00 to 100.00 with a mean of 76.48 and a standard deviation of 34.17
```

You can also directly apply many NumPy functions to datasets or DataArrays. For example, to find out how many unique bands the data contains, use [np.unique](https://numpy.org/doc/stable/reference/generated/numpy.unique.html):

```python Finding unique values theme={"system"}
import numpy as np
print("Sensors:", np.unique(satellite_data.bands))
```

```plaintext Output theme={"system"}
Sensors: [12]
```

## Reading and writing files

Xarray provides a simple method for saving and loading datasets from files. This is useful for sharing your data or storing it for future use. Xarray supports many different file formats, including NetCDF, Zarr, GRIB, and more. For a complete list of supported formats, refer to the [official documentation page](https://docs.xarray.dev/en/stable/user-guide/io.html).

To save the example dataset as a NetCDF file:

<Note>
  You may need to install the `netcdf4` package first.
</Note>

```python Saving a dataset to a file theme={"system"}
satellite_data.to_netcdf("example_satellite_data.nc")
```

This creates a file named `example_satellite_data.nc` in your current directory. You can then load this file back into memory with:

```python Loading a dataset from a file theme={"system"}
import xarray as xr

satellite_data = xr.open_dataset("example_satellite_data.nc")
```

If you would like to follow along with the examples in this section, you can download the example dataset as a NetCDF file [here][example_satellite_data.nc].

## Further reading

This section covers only a few common use cases for Xarray. The library offers many more functions and features. For more information, please see the [Xarray documentation](https://xarray.pydata.org/en/stable/) or explore the [Xarray Tutorials](https://tutorial.xarray.dev/intro.html).

Some useful capabilities not covered in this section include:

<Columns>
  <Card title="Grouping data" icon="layer-group" href="https://docs.xarray.dev/en/stable/user-guide/groupby.html" />

  <Card title="Computation" icon="calculator" href="https://docs.xarray.dev/en/stable/user-guide/computation.html" />

  <Card title="Time series specific functionality" icon="clock" href="https://docs.xarray.dev/en/stable/user-guide/time-series.html" />

  <Card title="Interpolation" icon="arrows-left-right" href="https://docs.xarray.dev/en/latest/user-guide/interpolation.html" />

  <Card title="Plotting" icon="chart-bar" href="https://docs.xarray.dev/en/latest/user-guide/plotting.html" />
</Columns>


# Automations
Source: https://docs.tilebox.com/workflows/automations/automations

Trigger workflow jobs automatically in response to external events such as schedules or file changes, enabling near-real-time data processing pipelines.

## Introduction

Tilebox Workflows can execute jobs in two ways: a one-time execution triggered by a user, typically a batch processing, and near-real-time execution based on specific external events.
By defining trigger conditions, you can automatically submit jobs based on external events.

Tilebox Workflows currently supports the following trigger conditions:

<Columns>
  <Card title="Cron Triggers" icon="clock" href="/workflows/automations/cron">
    Trigger jobs based on a Cron schedule.
  </Card>

  <Card title="Storage Event Triggers" icon="right-to-line" href="/workflows/automations/storage-events">
    Trigger jobs after objects are created or modified in a storage location such as a cloud bucket.
  </Card>
</Columns>

<Info>
  Dataset Event Triggers, which will trigger jobs when new data points are ingested into a Tilebox dataset, are on the roadmap. Stay tuned for updates.
</Info>

## Automations

To create a trigger, define a special task that serves as a prototype. In response to a trigger condition met, this task will be submitted as a new job. Such tasks are referred to as automations.

Each automation has a [task identifier](/workflows/concepts/tasks#task-identifiers), a [version](/workflows/concepts/tasks#semantic-versioning), and [input parameters](/workflows/concepts/tasks#input-parameters), just like regular tasks.
Automations also automatically provide a special `trigger` attribute that contains information about the event that initiated the task's execution.

## Automation Client

The Tilebox Workflows clients include sub-clients for inspecting automations and storage locations. Python also includes helpers for registering automation prototypes from task classes.

### Listing Registered Automations

To list all registered automations, use the automation client.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  automations = client.automations()

  automations = automations.all()
  print(automations)
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := workflows.NewClient()

  automations, err := client.Automations.List(ctx)
  if err != nil {
      slog.ErrorContext(ctx, "failed to list automations", slog.Any("error", err))
      return
  }

  for _, automation := range automations {
      slog.InfoContext(ctx,
          "automation",
          slog.String("id", automation.ID.String()),
          slog.String("name", automation.Name),
          slog.Int("cron_triggers", len(automation.CronTriggers)),
          slog.Int("storage_event_triggers", len(automation.StorageEventTriggers)),
      )
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
[
    AutomationPrototype(
        name='Run MyCronTask every hour at 15 minutes past the hour',
        prototype=TaskSubmission(
            cluster_slug='dev-cluster',
            identifier=TaskIdentifier(name='MyCronTask', version='v0.0'),
            input=b'{"message": "Hello"},
            dependencies=[],
            display='MyCronTask',
            max_retries=0), 
        storage_event_triggers=[],
        cron_triggers=[CronTrigger(schedule='15 * * * *')],
    )
]
```

### Registering Automations

To register an automation from code, use the Python automation client's `create_*_automation` methods for each trigger type. You can also create and manage automations in the Tilebox Console. Refer to the documentation for each trigger type for more details.

<Columns>
  <Card title="Cron Triggers" icon="clock" href="/workflows/automations/cron" />

  <Card title="Storage Event Triggers" icon="right-to-line" href="/workflows/automations/storage-events" />
</Columns>

## Overview in the Tilebox Console

You can also use the Tilebox Console to manage automations. Visit [the automations section](https://console.tilebox.com/workflows/automations) to check it out.

<Frame>
  <img alt="Tilebox Workflows automations in the Tilebox Console" />

  <img alt="Tilebox Workflows automations in the Tilebox Console" />
</Frame>

You can also register new automations or edit and delete existing ones directly from the console.

<Frame>
  <img alt="Tilebox Workflows automations in the Tilebox Console" />

  <img alt="Tilebox Workflows automations in the Tilebox Console" />
</Frame>


# Cron triggers
Source: https://docs.tilebox.com/workflows/automations/cron

Schedule recurring workflow jobs using cron expressions so that tasks run automatically at defined intervals, without requiring manual job submission.

## Creating Cron tasks

Cron tasks run repeatedly on a specified [cron](https://en.wikipedia.org/wiki/Cron) schedule. Define the task that the automation submits, then register its implementation with a runner.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext
  from tilebox.workflows.automations import CronTask

  class MyCronTask(CronTask):
      message: str

      def execute(self, context: ExecutionContext) -> None:
          # self.trigger is an attribute of the CronTask class,
          # which contains information about the trigger event
          # that caused this task to be submitted as part of a job
          context.logger.info(
              "Cron task triggered",
              message=self.message,
              trigger_time=self.trigger.time,
          )
  ```

  ```go Go theme={"system"}
  type MyCronTask struct {
      Message string
  }

  func (t *MyCronTask) Execute(ctx context.Context) error {
      slog.InfoContext(ctx, "Cron task triggered", slog.String("message", t.Message))
      return nil
  }
  ```
</CodeGroup>

## Registering a cron trigger

After implementing a cron task, register it with one or more schedules. The Python SDK provides a registration helper, and you can also register cron automations from the Tilebox Console. Each matching schedule submits a new job containing one task instance derived from the cron task prototype.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()
cron_automation = automations.create_cron_automation(
    "my-cron-automation",  # name of the cron automation
    MyCronTask(message="World"),  # the task (and its input parameters) to run repeatedly
    cron_schedules=[
        "*/15 * * * 1-5",  # every 15 minutes on weekdays in UTC
        "CRON_TZ=Europe/Vienna 0 9 * * 1-5",  # 09:00 on weekdays in Vienna
        "@daily",  # every day at midnight UTC
        "CRON_TZ=Europe/Vienna @daily",  # every day at midnight in Vienna
    ],
)
```

<Tip>
  Use [crontab.guru](https://crontab.guru/) to check standard five-field expressions. Remove any `CRON_TZ=...` prefix before entering an expression there.
</Tip>

## Cron schedule syntax

Cron triggers accept standard five-field expressions. The fields specify the minute, hour, day of the month, month, and day of the week in that order.

```text theme={"system"}
┌───────────── minute (0–59)
│ ┌─────────── hour (0–23)
│ │ ┌───────── day of month (1–31)
│ │ │ ┌─────── month (1–12 or JAN–DEC)
│ │ │ │ ┌───── day of week (0–6 or SUN–SAT; 0 is Sunday)
│ │ │ │ │
* * * * *
```

Use `*` for every value, `,` for a list, `-` for a range, and `/` for a step. For example, `*/15 * * * *` runs every 15 minutes, while `0 9-17 * * 1-5` runs hourly from 09:00 through 17:00 on weekdays.

If both day of month and day of week contain specific values, a trigger runs when either field matches. For example, `0 9 1 * MON` runs at 09:00 on the first day of every month and on every Monday.

Schedules without an explicit timezone use UTC. Prefix a schedule with `CRON_TZ=<timezone>` to interpret it in an [IANA timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones):

```text theme={"system"}
CRON_TZ=Europe/Vienna 0 9 * * 1-5
```

This expression runs at 09:00 Vienna local time on weekdays. The corresponding UTC time changes automatically when Vienna enters or leaves daylight saving time.

<Warning>
  Timezone schedules follow traditional cron behavior during daylight saving transitions. A local time skipped when clocks move forward does not trigger. A local time that occurs twice when clocks move backward triggers twice.
</Warning>

### Schedule helpers

You can replace a five-field expression with one of these helpers. Helpers use UTC unless you add a `CRON_TZ` prefix.

| Helper                   | Cron expression | Meaning                                    |
| ------------------------ | --------------- | ------------------------------------------ |
| `@yearly` or `@annually` | `0 0 1 1 *`     | At midnight on January 1                   |
| `@monthly`               | `0 0 1 * *`     | At midnight on the first day of each month |
| `@weekly`                | `0 0 * * 0`     | At midnight every Sunday                   |
| `@daily` or `@midnight`  | `0 0 * * *`     | At midnight every day                      |
| `@hourly`                | `0 * * * *`     | At the start of every hour                 |

Timezone prefixes also work with helpers:

```text theme={"system"}
CRON_TZ=Europe/Vienna @daily
```

This schedule runs every day at midnight in Vienna.

### Schedule examples

| Schedule                            | Meaning                                                       |
| ----------------------------------- | ------------------------------------------------------------- |
| `0 * * * *`                         | At the start of every hour in UTC                             |
| `*/15 * * * *`                      | Every 15 minutes in UTC                                       |
| `30 13 * * 3`                       | Every Wednesday at 13:30 UTC                                  |
| `0 9 1 * MON`                       | At 09:00 UTC on every Monday and every first day of the month |
| `CRON_TZ=Europe/Vienna 0 9 * * 1-5` | At 09:00 Vienna time on weekdays                              |
| `@daily`                            | Every day at midnight UTC                                     |
| `CRON_TZ=Europe/Vienna @daily`      | Every day at midnight in Vienna                               |
| `@weekly`                           | Every Sunday at midnight UTC                                  |

## Starting a cron runner

Cron tasks run on any regular [runner](/workflows/concepts/runners) that has the task registered. Keep at least one such runner available to execute jobs submitted by the automation.

```python Python theme={"system"}
from tilebox.workflows import Client, Runner

client = Client()
runner = Runner(tasks=[MyCronTask])
runner.connect_to(client).run_forever()
```

If this runner runs continuously, its logs may resemble the following:

```plaintext Logs theme={"system"}
Cron task triggered message=World trigger_time=2023-09-25 16:12:00
Cron task triggered message=World trigger_time=2023-09-25 17:12:00
Cron task triggered message=World trigger_time=2023-09-25 18:12:00
Cron task triggered message=World trigger_time=2023-09-25 18:45:00
Cron task triggered message=World trigger_time=2023-09-25 19:12:00
```

## Inspecting in the Console

The [Tilebox Console](https://console.tilebox.com/workflows/automations) provides a straightforward way to inspect all registered Cron automations.

<Frame>
  <img alt="Tilebox Workflows automations in the Tilebox Console" />

  <img alt="Tilebox Workflows automations in the Tilebox Console" />
</Frame>

Use the console to view, edit, and delete the registered Cron automations.

You can also inspect registered cron triggers from the SDKs.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  automations = client.automations().all()

  for automation in automations:
      for trigger in automation.cron_triggers:
          print(automation.name, trigger.schedule)
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := workflows.NewClient()

  automations, err := client.Automations.List(ctx)
  if err != nil {
      slog.ErrorContext(ctx, "failed to list automations", slog.Any("error", err))
      return
  }

  for _, automation := range automations {
      for _, trigger := range automation.CronTriggers {
          slog.InfoContext(ctx,
              "cron trigger",
              slog.String("automation", automation.Name),
              slog.String("schedule", trigger.Schedule),
          )
      }
  }
  ```
</CodeGroup>

## Deleting Cron automations

To delete a registered Cron automation from Python, use `automations.delete`. You can also delete cron automations from the Tilebox Console. After deletion, no new jobs will be submitted by that Cron trigger. Past jobs already triggered will still remain queued.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()

# delete the automation as returned by create_cron_automation
automations.delete(cron_automation)

# or manually by id:
automations.delete("0190bafc-b3b8-88c4-008b-a5db044380d0")
```

## Submitting Cron jobs manually

In Python, you can submit Cron tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific trigger time using the `once` method.

<Note>
  Submitting a job with a Cron task using `once` immediately schedules the task, and a runner may pick it up and execute it. The trigger time set in the `once` method does not influence the execution time; it only sets the `self.trigger.time` attribute for the Cron task.
</Note>

```python Python theme={"system"}
from datetime import datetime, timezone

job_client = client.jobs()

# create a Cron task prototype
task = MyCronTask(message="Hello")

# submitting it directly won't work: raises ValueError:
# job_client.submit("manual-cron-job", task)

# instead trigger a cron task with the current time as the trigger time
job_client.submit("manual-cron-job", task.once())

# or specify a trigger time in the past or future
# irrespective of the trigger time, the task will always be scheduled to run immediately
job_client.submit(
    "manual-cron-job", 
    task.once(datetime(2030, 12, 12, 15, 15, tzinfo=timezone.utc))
)
```


# Storage Event Triggers
Source: https://docs.tilebox.com/workflows/automations/storage-events

Trigger workflow jobs automatically whenever new objects are created or modified in a storage bucket, enabling event-driven data processing pipelines.

## Creating a Storage Event Task

Storage Event Tasks are automations triggered when objects are created or modified in a [storage location](#storage-locations).
In Python, use `tilebox.workflows.automations.StorageEventTask` as your task base class instead of the regular `tilebox.workflows.Task`. In Go, register an executable task with the same task identifier as the automation prototype.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext
  from tilebox.workflows.automations import StorageEventTask, StorageEventType

  class LogObjectCreation(StorageEventTask):
      head_bytes: int

      def execute(self, context: ExecutionContext) -> None:
          # self.trigger is an attribute of the StorageEventTask class,
          # which contains information about the storage event that triggered this task
          if self.trigger.type == StorageEventType.CREATED:
              path = self.trigger.location
              context.logger.info("A new object was created", path=path)

              # trigger.storage is a storage client for interacting with the storage
              # location that triggered the event
              # using it, we can read the object to get its content as a bytes object
              data = self.trigger.storage.read(path)
              context.logger.info("Read object data", path=path, size_bytes=len(data))
              context.logger.info("Object preview", path=path, preview=data[:self.head_bytes].hex())
  ```

  ```go Go theme={"system"}
  type LogObjectCreation struct {
      HeadBytes int
  }

  func (t *LogObjectCreation) Execute(ctx context.Context) error {
      slog.InfoContext(ctx, "storage event task triggered", slog.Int("head_bytes", t.HeadBytes))
      return nil
  }
  ```
</CodeGroup>

## Storage Locations

Storage Event tasks are triggered when objects are created or modified in a storage location. This location can be a cloud storage bucket or a local file system. Tilebox supports the following storage locations:

<Columns>
  <Card title="Google Cloud Storage" icon="google" />

  <Card title="Amazon S3" icon="aws" />

  <Card title="Local File System" icon="folder-open" />
</Columns>

### Registering a Storage Location

To make a storage location available within Tilebox workflows, it must be registered first. This involves specifying the location and setting up a notification system that forwards events to Tilebox, enabling task triggering. The setup varies depending on the storage location type.

For example, a GCP storage bucket is integrated by setting up a [PubSub Notification with a push subscription](https://cloud.google.com/storage/docs/pubsub-notifications). A local file system requires installing a filesystem watcher. To set up a storage location registered with Tilebox, please [get in touch](mailto:engineering@tilebox.com).

### Listing Available Storage Locations

To list all available storage locations, use the automation client.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  automations_client = client.automations()

  storage_locations = automations_client.storage_locations()
  print(storage_locations)
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := workflows.NewClient()

  storageLocations, err := client.Automations.ListStorageLocations(ctx)
  if err != nil {
      slog.ErrorContext(ctx, "failed to list storage locations", slog.Any("error", err))
      return
  }

  for _, location := range storageLocations {
      slog.InfoContext(ctx,
          "storage location",
          slog.String("id", location.ID.String()),
          slog.String("location", location.Location),
          slog.String("type", location.Type.String()),
      )
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
[
    StorageLocation(
        location="gcp-project:gcs-bucket-fab3fa2",
        type=StorageType.GCS,
    ),
    StorageLocation(
        location="s3-bucket-263af15",
        type=StorageType.S3,
    ),
    StorageLocation(
        location='/path/to/a/local/folder',
        type=StorageType.FS,
    ),
]
```

### Reading Files from a Storage Location

Once a storage location is registered, you can read files from it using the `read` method on the storage client.

```python Python theme={"system"}
gcs_bucket = storage_locations[0]
s3_bucket = storage_locations[1]
local_folder = storage_locations[2]

gcs_object = gcs_bucket.read("my-object.txt")
s3_object = s3_bucket.read("my-object.txt")
local_object = local_folder.read("my-object.txt")
```

<Note>
  The `read` method instantiates a client for the specific storage location. This requires that
  the storage location is accessible by a runner and may require credentials for cloud storage
  or physical/network access to a locally mounted file system.
</Note>

<Tip>
  To set up authentication and enable access to a GCS storage bucket, check out the [Google Client docs for authentication](https://cloud.google.com/docs/authentication/client-libraries#python).
</Tip>

## Registering a Storage Event Trigger

After implementing a Storage Event task, register it to trigger each time a storage event occurs. The Python SDK provides a registration helper, and you can also register storage-event automations from the Tilebox Console.
This registration submits a new job consisting of a single task instance derived from the registered Storage Event task prototype.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()
storage_event_automation = automations.create_storage_event_automation(
    "log-object-creations",  # name of the storage event automation
    LogObjectCreation(head_bytes=20),  # the task (and its input parameters) to run repeatedly
    triggers=[
        # you can specify a glob pattern:
        # run every time a .txt file is created anywhere in the gcs bucket
        (gcs_bucket, "**.txt"),  
    ],
)
```

<Tip>
  The syntax for specifying glob patterns follows [Standard Wildcards](https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm).
  Additionally, you can use `**` as a super-asterisk, a matching operator not sensitive to slash separators.
</Tip>

Here are some examples of valid glob patterns:

| Pattern     | Matches                                                                      |
| ----------- | ---------------------------------------------------------------------------- |
| `*.ext`     | Any file ending in `.ext` in the root directory                              |
| `**/*.ext`  | Any file ending in `.ext` in any subdirectory, but not in the root directory |
| `**.ext`    | Any file ending in `.ext` in any subdirectory, including the root directory  |
| `folder/*`  | Any file directly in a `folder` subdirectory                                 |
| `folder/**` | Any file directly or recursively part of a `folder` subdirectory             |
| `[a-z].txt` | Matches `a.txt`, `b.txt`, etc.                                               |

## Start a Storage Event runner

With the Storage Event automation registered, a job is submitted whenever a storage event occurs. But unless a [runner](/workflows/concepts/runners) is available to execute the Storage Event task the submitted jobs remain in a task queue.
Once an [eligible runner](/workflows/concepts/runners#task-selection) becomes available, all jobs in the queue are executed.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client, Runner

  client = Client()
  runner = Runner(tasks=[LogObjectCreation])
  runner.connect_to(client).run_forever()
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := workflows.NewClient()

  runner, err := client.NewTaskRunner(ctx)
  if err != nil {
      slog.ErrorContext(ctx, "failed to create task runner", slog.Any("error", err))
      return
  }

  if err := runner.RegisterTasks(&LogObjectCreation{}); err != nil {
      slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err))
      return
  }

  if err := runner.RunForever(ctx); err != nil {
      slog.ErrorContext(ctx, "task runner stopped", slog.Any("error", err))
  }
  ```
</CodeGroup>

### Triggering an Event

Creating an object in the bucket where the task is registered results in a job being submitted:

```bash Creating an object theme={"system"}
echo "Hello World" > my-object.txt
gcloud storage cp my-object.txt gs://gcs-bucket-fab3fa2
```

Inspecting the runner output reveals that the job was submitted and the task executed:

```plaintext Output theme={"system"}
2024-09-25 16:51:45,621 INFO A new object was created: my-object.txt
2024-09-25 16:51:45,857 INFO The object's file size is 12 bytes
2024-09-25 16:51:45,858 INFO The object's first 20 bytes are: b'Hello World\n'
```

## Inspecting in the Console

The [Tilebox Console](https://console.tilebox.com/workflows/automations) provides an easy way to inspect all registered storage event automations.

<Frame>
  <img alt="Tilebox Workflows automations in the Tilebox Console" />

  <img alt="Tilebox Workflows automations in the Tilebox Console" />
</Frame>

## Deleting Storage Event automations

To delete a registered storage event automation from Python, use `automations.delete`. You can also delete storage-event automations from the Tilebox Console. After deletion, no new jobs will be submitted by the storage event trigger. Past jobs already triggered will still remain queued.

```python Python theme={"system"}
from tilebox.workflows import Client

client = Client()
automations = client.automations()

# delete the automation as returned by create_storage_event_automation
automations.delete(storage_event_automation)

# or manually by id:
automations.delete("0190bafc-b3b8-88c4-008b-a5db044380d0")
```

## Submitting Storage Event jobs manually

In Python, you can submit Storage Event tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific storage location and object name using the `once` method.

```python Python theme={"system"}
job_client = client.jobs()

task = LogObjectCreation(head_bytes=20)

# submitting it directly won't work; raises ValueError:
# job_client.submit("manual-storage-event-job", task)

# instead, we specify a trigger condition, and submit a job manually
job_client.submit(
    "manual-storage-event-job",
    # simulate an event that occurred in the gcs bucket for the object "my-object.txt"
    task.once(gcs_bucket, "my-object.txt"),
)
```


# Cluster Deployments
Source: https://docs.tilebox.com/workflows/build-and-deploy/cluster-deployments

Understand how workflow release deployments map releases to clusters and control what release runners can execute.

Deploying a workflow release creates a cluster deployment. A cluster deployment maps one workflow release to one cluster, which controls what workflow code release runners on that cluster can load and execute.

Creating, updating, or removing a deployment does not submit a job, start a runner, delete a release, or change past jobs. Existing release runners for that cluster observe the deployment change and update which workflow code they can execute.

## Deployment model

A workflow can have many releases, and each release can be deployed to one or more clusters. A cluster can also have releases from multiple workflows deployed at the same time. Release runners on that cluster load the deployed releases and execute compatible tasks when matching jobs are submitted.

For a single workflow, a cluster has one active release deployment at a time. Updating the deployment changes which release new compatible task executions use on that cluster.

## Default and explicit clusters

Every team has a default cluster. If a deployment command does not specify a cluster or target, the default cluster is used. This is useful for quick tests when you do not need separate development or production clusters.

Explicit clusters are useful when environments should run different workflow code. For example, a development cluster can run the latest release while a production cluster keeps running a release that has already been tested.

Use explicit release IDs for production-like deployments when you need to know exactly which release is active. Use latest-release deployment only when that behavior is acceptable for the environment.

## Targets

Targets are named cluster groups defined in `tilebox.workflow.toml`. They let a deployment refer to a logical environment instead of repeating cluster slugs.

```toml theme={"system"}
[targets.dev]
clusters = ["dev-cluster"]

[targets.production]
clusters = ["prod-a", "prod-b"]
```

A target can contain one cluster or multiple clusters. The same release is deployed to each cluster in the target.

## Release runners and task execution

A release runner runs in an environment you control and watches one cluster. It loads workflow code from releases deployed to that cluster, starts Python workflow runtime processes through the Tilebox command-line tool, and updates its task set when deployments change.

Deployment alone is not enough for execution. A compatible release runner must be running for the cluster, and submitted tasks must match task identifiers and compatible versions from one of the deployed releases.

<Note>
  You still choose where the release runner process runs. Tilebox manages release selection through cluster deployments, while your environment provides the compute process that runs `tilebox runner start`.
</Note>

## Job and deployment clusters must match

When you submit a job, its root task is submitted to a cluster. For a release runner to execute that task, the job cluster, release deployment cluster, and release runner cluster must match.

If a job stays queued, check the cluster first. A task submitted to `dev-cluster` is not claimed by a release runner on `staging-cluster`, even if the same workflow release is deployed somewhere else.

## Removing deployments

Removing a release deployment removes the active release mapping from the selected cluster or target. It does not delete the workflow, release, artifact, or past jobs.

Release runners update their task set after a deployment is removed. They stop accepting new tasks that require the removed release, but already completed jobs and release records remain available for inspection.

## Related documentation

<Columns>
  <Card title="Release lifecycle" icon="box-open" href="/workflows/build-and-deploy/releases">
    Understand how releases are built, checked, published, deployed, and used by release runners.
  </Card>

  <Card title="Workflow configuration" icon="file-code" href="/workflows/build-and-deploy/workflow-configuration">
    Define deployment targets in `tilebox.workflow.toml`.
  </Card>

  <Card title="Clusters" icon="circle-nodes" href="/workflows/concepts/clusters">
    Learn how clusters determine where workflow tasks can run.
  </Card>

  <Card title="Runners" icon="list-check" href="/workflows/concepts/runners">
    Learn how release runners load deployed releases and execute compatible tasks.
  </Card>
</Columns>


# Project Structure
Source: https://docs.tilebox.com/workflows/build-and-deploy/project-structure

Initialize and structure a Python workflow project so releases can be built reproducibly and release runners can discover and execute its tasks.

A Python workflow project contains task classes, a `Runner` definition, and a `tilebox.workflow.toml` file. The Tilebox command-line tool uses these files to build a workflow release, discover the tasks it can execute, and make the release available to release runners after deployment.

<Tip>
  This project structure and `tilebox.workflow.toml` are required for [release runners](/workflows/concepts/runners#release-runners). For direct runners, you can organize your code however you want, as long as your runner process can import and register the tasks it executes. The same structure can still be useful when you want a small, importable workflow package.
</Tip>

Keep the project small and importable from its root. Release builds import the configured runner object, check that the Python runtime starts, and package the selected files into an immutable artifact.

## Scaffold a workflow project

For a new Python workflow project, use the CLI to create the Tilebox workflow and scaffold the local files.

```bash theme={"system"}
tilebox workflow init --name "Scene QA"
```

The `--name` flag is optional. When omitted, the command derives the local project slug from the current directory name. The command converts the name to a slug, truncates it to at most 40 characters, creates the remote workflow, and writes the API-returned workflow slug to `tilebox.workflow.toml`.

`tilebox workflow init` creates `tilebox.workflow.toml`, `pyproject.toml`, and `runner.py`, adds the `tilebox` Python dependency, and runs `uv sync` to create the local environment and `uv.lock` file. It requires `uv` on `PATH` and aborts without changing the directory if any of `tilebox.workflow.toml`, `pyproject.toml`, `runner.py`, or `uv.lock` already exists.

The generated project is intentionally small. Edit `runner.py` directly for prototypes, or move task code into a package as the workflow grows.

## Manual project structure

Use a layout where task code and the runner definition are importable from the project root.

<Tree>
  <Tree.Folder name="my-workflow">
    <Tree.File name="pyproject.toml" />

    <Tree.File name="uv.lock" />

    <Tree.File name="tilebox.workflow.toml" />

    <Tree.Folder name="my_workflow">
      <Tree.File name="__init__.py" />

      <Tree.File name="tasks.py" />

      <Tree.File name="runner.py" />
    </Tree.Folder>
  </Tree.Folder>
</Tree>

## Define the Python project

Use a minimal `pyproject.toml` with the dependencies your workflow needs. For this example, only the Tilebox Python package is required.

```toml pyproject.toml theme={"system"}
[project]
name = "my-workflow"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
  "tilebox",
]
```

Create the package directory and an empty `__init__.py` file so Python can import `my_workflow.runner` from the project root.

```bash theme={"system"}
mkdir -p my_workflow
touch my_workflow/__init__.py
uv lock
```

## Define tasks

Put task classes in a module that can be imported during release validation.

```python my_workflow/tasks.py theme={"system"}
# my_workflow/tasks.py
from tilebox.workflows import ExecutionContext, Task


class ProcessScene(Task):
    scene_id: str

    @staticmethod
    def identifier() -> tuple[str, str]:
        return "tilebox.com/example/ProcessScene", "v1.0"

    def execute(self, context: ExecutionContext) -> None:
        context.current_task.display = f"ProcessScene({self.scene_id})"
        context.logger.info("Processing scene", scene_id=self.scene_id)
```

Use explicit identifiers for workflow code that will be published. A stable identifier lets existing jobs continue to run after refactors and compatible bug fixes.

## Define the runner

Create a module that exports a `Runner` object. This object defines the task registrations for the workflow, and release builds import it during validation.

```python my_workflow/runner.py theme={"system"}
# my_workflow/runner.py
from tilebox.workflows import Runner
from tilebox.workflows.cache import LocalFileSystemCache

from my_workflow.tasks import ProcessScene


runner = Runner(
    tasks=[ProcessScene],
    cache=LocalFileSystemCache(),
)
```

## Configure the workflow release

Point `tilebox.workflow.toml` at the exported `Runner` object and include the files required by the release runner.

```toml theme={"system"}
[workflow]
slug = "my-workflow"
root = "."
runner = "my_workflow.runner:runner"

[build]
include = [
  "pyproject.toml",
  "uv.lock",
  "my_workflow/**",
]
exclude = [
  ".venv/**",
  "**/__pycache__/**",
  "**/*.pyc",
  ".pytest_cache/**",
]
use_gitignore = true
```

The Tilebox command-line tool imports the runner object during `build-release` and `publish-release`, discovers its task identifiers, and records them in the workflow release. A release runner later loads the release artifact and invokes the Python runtime through the command-line tool.

## Keep release artifacts small

Include source code, lock files, and small configuration. Exclude local virtual environments, test caches, downloaded provider data, model checkpoints, generated outputs, and other large runtime artifacts.

If a task needs a large model or reference file, fetch it lazily at runtime and cache it in a deterministic runner-local path such as `~/.cache/tilebox/...`. The workflow should still work when a release runner starts with an empty cache.


# Release Lifecycle
Source: https://docs.tilebox.com/workflows/build-and-deploy/releases

Understand how workflow releases are built, checked, published, deployed, and used by release runners.

Workflow releases make workflow code reproducible. A release captures the files selected from a Python workflow project, the runtime entrypoint, and the task identifiers discovered from the configured runner. Release runners later use that release to execute compatible tasks on clusters that have the release deployed.

The release lifecycle separates code packaging from cluster rollout. Building and publishing create immutable release content. Deploying a release changes which workflow code release runners on a cluster can execute.

## Lifecycle overview

| Stage                   | What it does                                                                                | What changes                                                                          |
| ----------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Workflow initialization | Creates the long-lived workflow object and local project configuration.                     | Adds a workflow record and, when using `tilebox workflow init`, local scaffold files. |
| Local build             | Resolves files, creates a deterministic artifact, and checks the configured Python runtime. | Writes local build output and cache entries.                                          |
| Publishing              | Uploads the artifact when needed and creates an immutable release record.                   | Adds or reuses a workflow release.                                                    |
| Deployment              | Maps a release to one or more clusters.                                                     | Changes which release runners can execute the release.                                |
| Job execution           | Submits tasks that match the release's task identifiers and compatible versions.            | Creates or updates jobs and tasks.                                                    |

## Workflow object

A workflow is the stable object that releases belong to. It has a slug, such as `my-workflow`, which is referenced from `tilebox.workflow.toml`. The workflow object is not the executable code itself; it provides the long-lived name under which releases are published.

Create the workflow object once, then keep publishing new releases to it as the code changes. For new Python workflow projects, `tilebox workflow init` creates the workflow object and writes the returned slug into `tilebox.workflow.toml`. This gives release runners and deployment commands a stable workflow identity while each release remains immutable.

## Release artifact

The release artifact is the package that release runners download and execute. Tilebox assembles it from the files selected by the `[build]` section in `tilebox.workflow.toml`, after include patterns, exclude patterns, and `.gitignore` handling are applied.

Artifacts should contain source code, lock files, and small configuration. They should not contain downloaded source data, model checkpoints, generated outputs, local virtual environments, or task caches. If a workflow needs large runtime assets, the task code should fetch them at runtime and cache them in the runner environment.

## Runtime checks and task discovery

During a build, Tilebox starts the configured Python workflow runtime and imports the runner object or command from `tilebox.workflow.toml`. This checks that the artifact contains the files needed to start the workflow runtime.

The same step discovers the task identifiers registered by the workflow. Those identifiers become part of the release metadata and determine which submitted tasks a release runner can execute after the release is deployed.

## Publishing

Publishing turns checked local release content into a workflow release in Tilebox. The release record points to the artifact, the discovered task registrations, and the workflow slug it belongs to.

Publishing is idempotent for identical release content and artifact digests. If the same release already exists, Tilebox returns the existing release instead of creating another copy. This is useful in agent-assisted and automated workflows where the same build step may run more than once.

## Deployment

Publishing a release does not make release runners execute it. A release must be deployed to a cluster before release runners on that cluster load its workflow code and execute compatible tasks from it.

A deployment maps one release to one cluster. Different clusters can run different releases of the same workflow, which makes it possible to test a release in a development cluster before promoting it to production-like clusters.

## Compatibility and retries

Release compatibility matters when retrying existing jobs. A failed job can resume from failed tasks after you deploy a compatible fixed release to the same cluster. The task identifier name, major version, and input schema must remain compatible with the failed task that is being retried.

Use a new major task version and submit a new job when the task input schema or behavior is no longer compatible with tasks submitted before the change. This keeps old jobs reproducible while allowing breaking workflow changes to move forward safely.

## Related documentation

<Columns>
  <Card title="Project structure" icon="folder-tree" href="/workflows/build-and-deploy/project-structure">
    Structure a Python workflow project for workflow releases and release runners.
  </Card>

  <Card title="Workflow configuration" icon="file-code" href="/workflows/build-and-deploy/workflow-configuration">
    Configure `tilebox.workflow.toml`, build inputs, and deployment targets.
  </Card>

  <Card title="Deploy releases to clusters" icon="rocket" href="/workflows/build-and-deploy/cluster-deployments">
    Map workflow releases to clusters and run them with release runners.
  </Card>

  <Card title="Workflow release concepts" icon="box-open" href="/workflows/concepts/workflow-releases">
    Learn how workflows, releases, artifacts, and cluster deployments fit together.
  </Card>
</Columns>


# Workflow Configuration
Source: https://docs.tilebox.com/workflows/build-and-deploy/workflow-configuration

Understand the tilebox.workflow.toml fields that define workflow identity, release contents, runtime entrypoints, and deployment targets.

`tilebox.workflow.toml` describes how a Python workflow project becomes a workflow release. It identifies the Tilebox workflow, defines which files belong in the release artifact, selects the runtime entrypoint, and optionally names cluster targets for deployments.

The Tilebox command-line tool searches upward from the current directory for the nearest configuration file. This lets release commands run from the project root or from subdirectories inside the project.

<Tip>
  For new projects, run `tilebox workflow init` to create `tilebox.workflow.toml`, `pyproject.toml`, and `runner.py`. Edit the generated configuration when you need package layouts, deployment targets, or custom build includes.
</Tip>

## Minimal configuration

```toml theme={"system"}
[workflow]
slug = "my-workflow"
root = "."
runner = "my_workflow.runner:runner"

[build]
include = [
  "pyproject.toml",
  "uv.lock",
  "my_workflow/**",
]
exclude = [
  ".venv/**",
  "**/__pycache__/**",
  "**/*.pyc",
  ".pytest_cache/**",
]
use_gitignore = true
```

## Workflow section

The `[workflow]` section identifies the workflow and tells the Tilebox command-line tool how to start the Python workflow runtime during release checks and release execution.

| Field     | Required                     | Description                                                                                                                                             |
| --------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `slug`    | Yes                          | Stable workflow slug, such as `my-workflow`. Releases are published under this workflow.                                                                |
| `root`    | No                           | Project root for build paths. Defaults to `"."`.                                                                                                        |
| `runner`  | One of `runner` or `command` | Python module/object path to a `Runner` object. The Tilebox command-line tool runs it with `uv run python -m tilebox.workflows.runner <module:object>`. |
| `command` | One of `runner` or `command` | Custom worker command as an array of strings. Use this only when the standard runner object path does not fit your runtime.                             |

Set exactly one of `runner` or `command`.

## Build section

The `[build]` section controls which files are included in the release artifact.

| Field           | Required | Description                                                         |
| --------------- | -------- | ------------------------------------------------------------------- |
| `include`       | Yes      | Glob patterns, relative to `[workflow].root`, for files to include. |
| `exclude`       | No       | Glob patterns to exclude from the included file set.                |
| `use_gitignore` | No       | Whether to apply `.gitignore`. Defaults to `true`.                  |

Include lock files and source files so release runners resolve the same dependencies that were tested during release validation. Exclude local environments, generated files, caches, and large runtime assets.

## Targets

Targets name reusable cluster groups. They are optional, but useful when the same release should be deployed to a known set of development, staging, or production clusters.

```toml theme={"system"}
[targets.dev]
clusters = ["dev-cluster"]

[targets.production]
clusters = ["prod-a", "prod-b"]
```

A target can contain one cluster or multiple clusters. Deployment commands can refer to the target name instead of listing the same cluster slugs every time. For production-like deployments, use explicit release IDs so the active release is traceable.

## Validation rules

The Tilebox command-line tool checks configuration before building or publishing a release:

* `workflow.slug` is required.
* Exactly one of `workflow.runner` or `workflow.command` is required.
* `build.include` must contain at least one pattern.
* Unknown TOML keys fail configuration loading.
* Build paths must stay within the configured workflow root.


# Clusters
Source: https://docs.tilebox.com/workflows/concepts/clusters

Use clusters to group runners, target job execution, and control where workflow releases are deployed.

A cluster determines where workflow tasks can run. When you submit a [job](/workflows/concepts/jobs), its root [task](/workflows/concepts/tasks) is submitted to a cluster, and only [runners](/workflows/concepts/runners) assigned to that cluster can claim it.

By default, subtasks are submitted to the same cluster as their parent task. Task code can submit individual subtasks to other clusters when a workflow needs to execute across different environments.

[Workflow releases](/workflows/concepts/workflow-releases) can be deployed to individual clusters. This lets [release runners](/workflows/concepts/runners#release-runners) automatically load workflow code and execute compatible tasks from those releases.

## Use Cases

Use clusters to organize [runners](/workflows/concepts/runners) and workflow deployments into logical groups, which can help with:

* Targeting specific runners for a particular job
* Reserving a group of runners for specific purposes, such as running certain types of batch jobs
* Setting up different clusters for different environments (like development and production)
* Deploying different workflow releases to development, staging, or production clusters

Even within the same cluster, runners may have different capabilities. A direct runner advertises the tasks registered in its own process. A release runner advertises tasks from the workflow releases currently deployed to that cluster.

## Cluster deployments

A workflow release deployment maps one workflow release to one cluster. Deploying a release does not submit a job. It only makes the release available to release runners on that cluster.

A release runner can run multiple deployed releases for the same cluster. While it runs, it polls cluster deployment state and updates its task registrations when releases are deployed, updated, or removed.

### Adding runners to a cluster

You can add direct runners to a cluster by specifying the [cluster's slug](#cluster-slug) when starting the runner from your SDK code. You can add release runners to a cluster with `tilebox runner start --cluster <cluster-slug>`.
Each runner must always be assigned to a cluster. If no cluster is specified, Tilebox uses the default cluster.

## Default Cluster

Each team has a default cluster that is automatically created for them.
This cluster is used when no cluster is specified when [starting a runner](/workflows/concepts/runners), [deploying a release](/workflows/build-and-deploy/cluster-deployments), or [submitting a job](/workflows/concepts/jobs).
This is useful when you are just getting started and don't need to create any custom clusters yet.

## Managing Clusters

Before starting a runner or submitting a job to a custom cluster, create the cluster. You can also list, fetch, and delete clusters as needed. The following sections explain how to do this.

To manage clusters, first instantiate a cluster client using the `clusters` method in the workflows client.

<CodeGroup title="Instantiating a Clusters Client">
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  clusters = client.clusters()
  ```

  ```go Go theme={"system"}
  import "github.com/tilebox/tilebox-go/workflows/v1"

  client := workflows.NewClient()
  clusterClient := client.Clusters
  ```
</CodeGroup>

### Creating a Cluster

To create a cluster, use the `create` method on the cluster client and provide a name for the cluster.

<CodeGroup title="Creating a Cluster">
  ```python Python theme={"system"}
  cluster = clusters.create("testing")
  print(cluster)
  ```

  ```go Go theme={"system"}
  cluster := client.Clusters.Create("testing")
  fmt.Println(cluster)
  ```
</CodeGroup>

<CodeGroup title="Output">
  ```plaintext Python theme={"system"}
  Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing')
  ```

  ```go Go theme={"system"}
  &{testing-CvufcSxcC9SKfe testing}
  ```
</CodeGroup>

### Cluster Slug

Each cluster has a unique identifier, combining the cluster's name and an automatically generated identifier.
Use this slug to reference the cluster for other operations, like submitting a job or subtasks.

### Listing Clusters

To list all available clusters, use the `all` method:

<CodeGroup title="Listing Clusters">
  ```python Python theme={"system"}
  all_clusters = clusters.all()
  print(all_clusters)
  ```

  ```go Go theme={"system"}
  clusters, err := client.Clusters.List(ctx)
  if err != nil {
      slog.Error("failed to list clusters", slog.Any("error", err))
      return
  }

  for _, cluster := range clusters {
      fmt.Println(cluster)
  }
  ```
</CodeGroup>

<CodeGroup title="Output">
  ```plaintext Python theme={"system"}
  [Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing'),
  Cluster(slug='production-EifhUozDpwAJDL', display_name='Production')]
  ```

  ```go Go theme={"system"}
  &{testing-CvufcSxcC9SKfe testing}
  &{production-EifhUozDpwAJDL Production}
  ```
</CodeGroup>

### Fetching a Specific Cluster

To fetch a specific cluster, use the `find` method and pass the cluster's slug:

<CodeGroup title="Fetching a Cluster">
  ```python Python theme={"system"}
  cluster = clusters.find("testing-CvufcSxcC9SKfe")
  print(cluster)
  ```

  ```go Go theme={"system"}
  cluster, err := client.Clusters.Get(ctx, "testing-CvufcSxcC9SKfe")
  if err != nil {
  	slog.Error("failed to get cluster", slog.Any("error", err))
  	return
  }
  fmt.Println(cluster)
  ```
</CodeGroup>

<CodeGroup title="Output">
  ```plaintext Python theme={"system"}
  Cluster(slug='testing-CvufcSxcC9SKfe', display_name='testing')
  ```

  ```go Go theme={"system"}
  &{testing-CvufcSxcC9SKfe testing}
  ```
</CodeGroup>

### Deleting a Cluster

To delete a cluster, use the `delete` method and pass the cluster's slug:

<CodeGroup title="Deleting a Cluster">
  ```python Python theme={"system"}
  clusters.delete("testing-CvufcSxcC9SKfe")
  ```

  ```go Go theme={"system"}
  err := client.Clusters.Delete(ctx, "testing-CvufcSxcC9SKfe")
  ```
</CodeGroup>

## Jobs Across Different Clusters

When [submitting a job](/workflows/concepts/jobs), you need to specify which cluster the job's root task should be executed on.
This allows you to direct the job to a specific set of runners.
By default, all sub-tasks within a job are also submitted to the same cluster, but this can be overridden to submit sub-tasks to different clusters if needed.
See the example below for a job that spans across multiple clusters.

<CodeGroup title="Multi-Cluster Workflow Example">
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext, Client

  class MultiCluster(Task):
      def execute(self, context: ExecutionContext) -> None:
          # this submits a task to the same cluster as the one currently executing this task
          same_cluster = context.submit_subtask(DummyTask())
          
          other_cluster = context.submit_subtask(
              DummyTask(),
              # this task runs only on a runner in the "other-cluster" cluster
              cluster="other-cluster-As3dcSb3D9SAdK",
              # dependencies can be specified across clusters
              depends_on=[same_cluster],
          )

  class DummyTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          pass

  # submit a job to the "testing" cluster
  client = Client()
  job_client = client.jobs()
  job = job_client.submit(
      "my-job",
      MultiCluster(),
      cluster="testing-CvufcSxcC9SKfe",
  )
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"

  	"github.com/tilebox/tilebox-go/workflows/v1"
      "github.com/tilebox/tilebox-go/workflows/v1/subtask"
  )

  type MultiCluster struct{}

  func (t *MultiCluster) Execute(ctx context.Context) error {
  	// this submits a task to the same cluster as the one currently executing this task
  	sameCluster, err := workflows.SubmitSubtask(ctx, &DummyTask{})
  	if err != nil {
  		return err
  	}

  	otherCluster, err := workflows.SubmitSubtask(
  		ctx,
  		&DummyTask{},
  		// this task runs only on a runner in the "other-cluster" cluster
  		subtask.WithClusterSlug("other-cluster-As3dcSb3D9SAdK"),
  		// dependencies can be specified across clusters
  		subtask.WithDependencies(sameCluster),
  	)
  	if err != nil {
  		return err
  	}

  	_ = otherCluster
  	return nil
  }

  type DummyTask struct{}

  func main() {
      ctx := context.Background()
  	client := workflows.NewClient()

  	// submit a job to the "testing" cluster
  	_, _ = client.Jobs.Submit(
  		ctx,
  		"my-job",
  		[]workflows.Task{
  			&MultiCluster{},
  		},
          job.WithClusterSlug("testing-CvufcSxcC9SKfe"),
  	)
  }
  ```
</CodeGroup>

This workflow requires at least two runners to complete. One must be in the "testing" cluster, and the other must be in the "other-cluster" cluster.
If no runners are available in the "other-cluster," the task submitted to that cluster will remain queued until a runner is available.
It won't execute on a runner in the "testing" cluster, even if that runner has the `DummyTask` registered.


# Jobs
Source: https://docs.tilebox.com/workflows/concepts/jobs

Submit workflow jobs, target clusters, inspect job state, visualize execution, cancel jobs, and retry failed work.

A job is one execution of a workflow, starting from a root [task](/workflows/concepts/tasks) with concrete input values. As the root task runs, it can submit subtasks, creating the task graph that belongs to the same job.

When you submit a job, its root task is assigned to a [cluster](/workflows/concepts/clusters). Compatible [runners](/workflows/concepts/runners) execute tasks as they become eligible, and Tilebox updates job state from submission through completion, failure, cancellation, or retry.

## Submission

To execute a [task](/workflows/concepts/tasks), it must be initialized with concrete inputs and submitted as a job. The task will then run within the context of the job, and if it generates sub-tasks, those will also execute as part of the same job.

After submitting a job, the root task is scheduled for execution, and any [eligible runner](/workflows/concepts/runners#task-selection) can pick it up and execute it.

First, instantiate a job client by calling the `jobs` method on the workflow client.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  job_client = client.jobs()
  ```

  ```go Go theme={"system"}
  import "github.com/tilebox/tilebox-go/workflows/v1"

  client := workflows.NewClient()
  jobClient := client.Jobs
  ```
</CodeGroup>

After obtaining a job client, submit a job using the [submit](/api-reference/python/tilebox.workflows/JobClient.submit) method. You need to provide a name for the job, an instance of the root [task](/workflows/concepts/tasks), and an optional [cluster](/workflows/concepts/clusters) to execute the root task on.

<CodeGroup>
  ```python Python theme={"system"}
  # import your own workflow
  from my_workflow import MyTask

  job = job_client.submit('my-job', MyTask("some", "parameters"))
  ```

  ```go Go theme={"system"}
  job, err := client.Jobs.Submit(ctx, "my-job",
      []workflows.Task{
          &MyTask{
              Some: "parameters",
          },
      },
  )
  if err != nil {
      slog.Error("Failed to submit job", slog.Any("error", err))
      return
  }
  ```
</CodeGroup>

Once a job is submitted, it's immediately scheduled for execution. The root task will be picked up and executed as soon as an [eligible runner](/workflows/concepts/runners#task-selection) is available.

## Retry Handling

[Tasks support retry handling](/workflows/concepts/tasks#retry-handling) for failed executions. This applies to the root task of a job as well, where you can specify the number of retries using the `max_retries` argument of the `submit` method.

<CodeGroup>
  ```python Python theme={"system"}
  from my_workflow import MyFlakyTask

  job = job_client.submit('my-job', MyFlakyTask(), max_retries=5)
  ```

  ```go Go theme={"system"}
  myJob, err := client.Jobs.Submit(ctx, "my-job",
      []workflows.Task{
          &MyFlakyTask{},
      },
      job.WithMaxRetries(5),
  )
  ```
</CodeGroup>

In this example, if `MyFlakyTask` fails, it will be retried up to five times before being marked as failed.

## Submitting to a specific cluster

Jobs default to running on the [default cluster](/workflows/concepts/clusters#default-cluster).
You can specify another cluster to run the root task on using the `cluster` argument of the `submit` method.

<CodeGroup>
  ```python Python theme={"system"}
  from my_workflow import MyFlakyTask

  job = job_client.submit('my-job', MyFlakyTask(), cluster="dev-cluster")
  ```

  ```go Go theme={"system"}
  myJob, err := client.Jobs.Submit(ctx, "my-job",
      []workflows.Task{
          &MyFlakyTask{},
      },
      job.WithClusterSlug("dev-cluster"),
  )
  ```
</CodeGroup>

Only runners listening on the specified cluster can pick up the task.

## Querying jobs

You can query jobs in a given time range using the `query` method on the job client.

<CodeGroup>
  ```python Python theme={"system"}
  jobs = job_client.query(("2025-01-01", "2025-02-01"))
  print(jobs)
  ```

  ```go Go theme={"system"}
  import (
    "time"
    workflows "github.com/tilebox/tilebox-go/workflows/v1"
    "github.com/tilebox/tilebox-go/workflows/v1/job"
    "github.com/tilebox/tilebox-go/query"
  )

  interval := query.NewTimeInterval(
    time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC),
    time.Date(2025, 2, 1, 0, 0, 0, 0, time.UTC),
  )

  jobs, err := workflows.Collect(client.Jobs.Query(ctx, 
      job.WithTemporalExtent(interval),
    ))
  if err != nil {
      slog.Error("Failed to query jobs", slog.Any("error", err))
      return
  }

  for _, job := range jobs {
      fmt.Println(job)
  }
  ```
</CodeGroup>

## Retrieving a specific job

When you submit a job, it's assigned a unique identifier that can be used to retrieve it later.

You can use the `find` method on the job client to get a job by its ID.

<CodeGroup title="Retrieving a Job by ID">
  ```python Python theme={"system"}
  job = job_client.submit('my-job', MyTask("some", "parameters"))
  print(job.id)  # 018dd029-58ca-74e5-8b58-b4f99d610f9a

  # Later, in another process or machine, retrieve job info
  job = job_client.find("018dd029-58ca-74e5-8b58-b4f99d610f9a")
  ```

  ```go Go theme={"system"}
  myJob, err := client.Jobs.Submit(ctx, "my-job",
      []workflows.Task{
          &helloworld.HelloTask{
              Some: "parameters",
          },
      },
  )
  if err != nil {
      slog.Error("Failed to submit job", slog.Any("error", err))
      return
  }

  // 018dd029-58ca-74e5-8b58-b4f99d610f9a
  slog.Info("Job submitted", slog.String("job_id", myJob.ID.String()))

  // Later, in another process or machine, retrieve job info
  job, err := client.Jobs.Get(ctx, uuid.MustParse("018dd029-58ca-74e5-8b58-b4f99d610f9a"))
  ```
</CodeGroup>

<Tip>
  `find` is also a useful tool for fetching a jobs state after a while, to check if it's still running or has already completed.
</Tip>

In interactive environments such as Jupyter notebooks, the job object also provides a rich display of the job's state and progress, if it's used as the last expression in a cell.

<Frame>
  <img alt="Job display in a Jupyter notebook" />

  <img alt="Job display in a Jupyter notebook" />
</Frame>

## States

Every Job is always in exactly one of the following states:

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Submitted </div>
  <div>The Job hasn't started yet, all it's tasks are queued and it wasn't canceled by the user.</div>
</div>

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Running </div>
  <div>At least one task of the job is currently running.</div>
</div>

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Started </div>
  <div>The job has started, some tasks are already `COMPUTED`, but others are still `QUEUED`, waiting for an [eligible runner](/workflows/concepts/runners#task-selection) to pick them up. However no task is currently `RUNNING`.</div>
</div>

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Completed </div>
  <div>The job has successfully completed. Every task of the job succeeded and is `COMPUTED`.</div>
</div>

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Failed </div>
  <div>At least one task of the job has failed, causing the execution of the remaining tasks to be halted. You can [retry](#retries) the job to resume execution from the point of failure.</div>
</div>

<div>
  <div><img alt="Job submitted" /><img alt="Job submitted" /> Canceled </div>
  <div>The job was canceled upon user request. You can [retry](#retries) the job to resume execution from the point of cancellation.</div>
</div>

<Tip>
  The state of a job is determined by the states of all it's tasks. For a list of possible task states, see the [task state](/workflows/concepts/tasks#task-states) documentation.
</Tip>

You can programmatically check the state of a job by inspecting it's `state` field.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows.data import JobState

  job = job_client.find("018dd029-58ca-74e5-8b58-b4f99d610f9a")

  print("Job is running:", job.state == JobState.RUNNING)
  ```

  ```go Go theme={"system"}
  job, err := client.Jobs.Get(ctx, uuid.MustParse("018dd029-58ca-74e5-8b58-b4f99d610f9a"))

  fmt.Println("Job is running:", job.State == workflows.JobRunning)
  ```
</CodeGroup>

```plaintext Output theme={"system"}
Job is running: True
```

## Visualization

Visualizing the execution of a job can be helpful. The Tilebox workflow orchestrator tracks all tasks in a job, including [sub-tasks](/workflows/concepts/tasks#task-composition-and-subtasks) and [dependencies](/workflows/concepts/tasks#dependencies). This enables the visualization of the execution of a job as a graph diagram.

<Note>
  `display` is designed for use in an [interactive environment](/sdks/python/sample-notebooks#interactive-environments) such as a Jupyter notebook. In non-interactive environments, use [visualize](/api-reference/python/tilebox.workflows/JobClient.visualize), which returns the rendered diagram as an SVG string.
</Note>

<Note>
  Visualization isn't supported in Go yet.
</Note>

<CodeGroup>
  ```python Python theme={"system"}
  job = job_client.find("some-job-id")  # or a recently submitted job
  # Then visualize it
  job_client.display(job)
  ```
</CodeGroup>

The following diagram represents the job execution as a graph. Each task is shown as a node, with edges indicating sub-task relationships. The diagram also uses color coding to display the state of each task.

<Frame>
  <img alt="Color coding of task states" />

  <img alt="Color coding of task states" />
</Frame>

Below is another visualization of a job currently being executed by multiple runners.

<Frame>
  <img alt="Job being executed by multiple runners" />

  <img alt="Job being executed by multiple runners" />
</Frame>

From the diagram, the following can be inferred:

* The root task, `MyTask`, has been executed, is marked as `COMPUTED` and submitted three sub-tasks.
* At least three runners are available, as three tasks currently are executed simultaneously.
* The `SubTask` that is still executing has not generated any sub-tasks yet, as sub-tasks are queued for execution only after the parent task finishes and becomes computed.
* The queued `DependentTask` requires the `LeafTask` to complete before it can be executed.

<Note>
  Job visualizations are meant for development and debugging. They are not suitable for large jobs with hundreds of tasks, as the diagrams may become too complex. Currently, visualizations are limited to jobs with a maximum of 200 tasks.
</Note>

### Customizing Task Display Names

The text representing a task in the diagram defaults to a tasks class name. You can customize this by modifying the `display` field of the `current_task` object in the task's execution context. The maximum length for a display name is 1024 characters, with any overflow truncated. Line breaks using `\n` are supported as well.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class RootTask(Task):
      num_subtasks: int

      def execute(self, context: ExecutionContext):
          context.current_task.display = f"Root({self.num_subtasks})"
          for i in range(self.num_subtasks):
              context.submit_subtask(SubTask(i))

  class SubTask(Task):
      index: int

      def execute(self, context: ExecutionContext):
          context.current_task.display = f"Leaf Nr. {self.index}"

  job = job_client.submit('custom-display-names', RootTask(3))
  job_client.display(job)
  ```

  ```go Go theme={"system"}
  type RootTask struct {
  	NumSubtasks int
  }

  func (t *RootTask) Execute(ctx context.Context) error {
  	err := workflows.SetTaskDisplay(ctx, fmt.Sprintf("Root(%d)", t.NumSubtasks))
  	if err != nil {
  		return fmt.Errorf("failed to set task display: %w", err)
  	}

  	for i := range t.NumSubtasks {
  		_, err := workflows.SubmitSubtask(ctx, &SubTask{Index: i})
  		if err != nil {
  			return fmt.Errorf("failed to submit subtask: %w", err)
  		}
  	}
  	return nil
  }

  type SubTask struct {
  	Index int
  }

  func (t *SubTask) Execute(ctx context.Context) error {
  	err := workflows.SetTaskDisplay(ctx, fmt.Sprintf("Leaf Nr. %d", t.Index))
  	if err != nil {
  		return fmt.Errorf("failed to set task display: %w", err)
  	}
  	return nil
  }

  // in main
  job, err := client.Jobs.Submit(ctx, "custom-display-names",
  	[]workflows.Task{&RootTask{
  		NumSubtasks: 3,
  	}},
  )
  ```
</CodeGroup>

<Frame>
  <img alt="Customize Tasks Display Names" />

  <img alt="Customize Tasks Display Names" />
</Frame>

## Cancellation

You can cancel a job at any time. When a job is canceled, no queued tasks will be picked up by runners and executed even if runners are idle. Tasks that are already being executed will finish their execution and not be interrupted. All sub-tasks spawned from such tasks after the cancellation will not be picked up by runners.

Use the `cancel` method on the job client to cancel a job.

<CodeGroup>
  ```python Python theme={"system"}
  job = job_client.submit('my-job', MyTask())
  # After a short while, the job gets canceled
  job_client.cancel(job)
  ```

  ```go Go theme={"system"}
  job, err := client.Jobs.Submit(ctx, "my-job",
      []workflows.Task{&MyTask{}},
  )
  if err != nil {
      slog.Error("Failed to submit job", slog.Any("error", err))
      return
  }

  // After a short while, the job gets canceled
  err = client.Jobs.Cancel(ctx, job.ID)
  ```
</CodeGroup>

<Note>
  A canceled job can be resumed at any time by [retrying](#retries) it.
</Note>

If any task in a job fails, the job is automatically canceled to avoid executing irrelevant tasks. Future releases will allow configuring this behavior for each task to meet specific requirements.

## Retries

If a task fails due to a bug or lack of resources, there is no need to resubmit the entire job. You can simply retry the job, and it will resume from the point of failure. This ensures that all the work that was already done up until the point of the failure isn't lost.

<Note>
  Future releases may introduce automatic retries for certain failure conditions, which can be useful for handling temporary issues.
</Note>

Below is an example of a failing job due to a bug in the task's implementation. The following workflow processes a list of movie titles and queries the [OMDb API](http://www.omdbapi.com/) for each movie's release date.

<CodeGroup>
  ```python Python theme={"system"}
  from urllib.parse import urlencode
  import httpx
  from tilebox.workflows import Task, ExecutionContext

  class MoviesStats(Task):
      titles: list[str]

      def execute(self, context: ExecutionContext) -> None:
          for title in self.titles:
              context.submit_subtask(PrintMovieStats(title))

  class PrintMovieStats(Task):
      title: str

      def execute(self, context: ExecutionContext) -> None:
          params = {"t": self.title, "apikey": "<OMDB API Key>"}
          url = "http://www.omdbapi.com/?" + urlencode(params)
          with context.tracer.span("fetch-movie-stats") as span:
              span.set_attribute("movie.title", self.title)
              response = httpx.get(url).json()
          # set the display name of the task to the title of the movie:
          context.current_task.display = response["Title"]
          context.logger.info(
              "Movie release date fetched",
              title=response["Title"],
              released=response["Released"],
          )
  ```

  ```go Go theme={"system"}
  package movie

  import (
  	"context"
  	"encoding/json"
  	"fmt"
  	"io"
  	"net/http"
  	"net/url"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type MoviesStats struct {
  	Titles []string
  }

  func (t *MoviesStats) Execute(ctx context.Context) error {
  	for _, title := range t.Titles {
  		_, err := workflows.SubmitSubtask(ctx, &PrintMovieStats{Title: title})
  		if err != nil {
  			return fmt.Errorf("failed to submit subtask: %w", err)
  		}
  	}
  	return nil
  }

  type Movie struct {
  	Title    *string `json:"Title"`
  	Released *string `json:"Released"`
  }

  type PrintMovieStats struct {
  	Title string
  }

  func (t *PrintMovieStats) Execute(ctx context.Context) error {
  	apiURL := fmt.Sprintf("http://www.omdbapi.com/?t=%s&apikey=%s", url.QueryEscape(t.Title), "<OMDB API Key>")
  	response, err := http.Get(apiURL)
  	if err != nil {
  		return fmt.Errorf("failed to fetch movie: %w", err)
  	}

  	defer response.Body.Close()
  	body, err := io.ReadAll(response.Body)
  	if err != nil {
  		return fmt.Errorf("failed to read response: %w", err)
  	}

  	var movie Movie
  	err = json.Unmarshal(body, &movie)
  	if err != nil {
  		return fmt.Errorf("failed to unmarshal response: %w", err)
  	}

  	// set the display name of the task to the title of the movie:
      err := workflows.SetTaskDisplay(ctx, *movie.Title)
  	if err != nil {
  		return fmt.Errorf("failed to set task display: %w", err)
  	}

  	fmt.Printf("%s was released on %s\n", *movie.Title, *movie.Released)
  	return nil
  }
  ```
</CodeGroup>

Submitting the workflow as a job reveals a bug in the `PrintMovieStats` task.

<CodeGroup>
  ```python Python theme={"system"}
  job = job_client.submit('movies-stats', MoviesStats([
      "The Matrix",
      "Shrek 2",
      "Tilebox - The Movie",
      "The Avengers",
  ]))

  job_client.display(job)
  ```

  ```go Go theme={"system"}
  job, err := client.Jobs.Submit(ctx, "movies-stats",
      []workflows.Task{&MoviesStats{
          Titles: []string{
              "The Matrix",
              "Shrek 2",
              "Tilebox - The Movie",
              "The Avengers",
          },
      }},
  )
  ```
</CodeGroup>

<Frame>
  <img alt="Job that failed due to a bug" />

  <img alt="Job that failed due to a bug" />
</Frame>

One of the `PrintMovieStats` tasks fails with a `KeyError`. This error occurs when a movie title is not found by the [OMDb API](http://www.omdbapi.com/), leading to a response without the `Title` and `Released` fields.

Task logs from the runners confirm this:

```plaintext Logs theme={"system"}
Movie release date fetched title="The Matrix" released="31 Mar 1999"
Movie release date fetched title="Shrek 2" released="19 May 2004"
ERROR: Task PrintMovieStats failed with exception: KeyError('Title')
```

The corrected version of `PrintMovieStats` is as follows:

<CodeGroup title="Fixed Task Implementation">
  ```python Python theme={"system"}
  class PrintMovieStats(Task):
      title: str

      def execute(self, context: ExecutionContext) -> None:
          params = {"t": self.title, "apikey": "<OMDB API Key>"}
          url = "http://www.omdbapi.com/?" + urlencode(params)
          with context.tracer.span("fetch-movie-stats") as span:
              span.set_attribute("movie.title", self.title)
              response = httpx.get(url).json()
          if "Title" in response and "Released" in response:
              context.current_task.display = response["Title"]
              context.logger.info(
                  "Movie release date fetched",
                  title=response["Title"],
                  released=response["Released"],
              )
          else:
              context.current_task.display = f"NotFound: {self.title}"
              context.logger.info("Movie release date not found", title=self.title)
  ```

  ```go Go theme={"system"}
  type PrintMovieStats struct {
  	Title string
  }

  func (t *PrintMovieStats) Execute(ctx context.Context) error {
  	url2 := fmt.Sprintf("http://www.omdbapi.com/?t=%s&apikey=%s", url.QueryEscape(t.Title), "<OMDB API Key>")
  	response, err := http.Get(url2)
  	if err != nil {
  		return fmt.Errorf("failed to fetch movie: %w", err)
  	}

  	defer response.Body.Close()
  	body, err := io.ReadAll(response.Body)
  	if err != nil {
  		return fmt.Errorf("failed to read response: %w", err)
  	}

  	var movie Movie
  	err = json.Unmarshal(body, &movie)
  	if err != nil {
  		return fmt.Errorf("failed to unmarshal response: %w", err)
  	}

  	if movie.Released != nil && movie.Title != nil {
  		err := workflows.SetTaskDisplay(ctx, *movie.Title)
  		if err != nil {
  			return fmt.Errorf("failed to set task display: %w", err)
  		}
  		fmt.Printf("%s was released on %s\n", *movie.Title, *movie.Released)
  	} else {
  		err := workflows.SetTaskDisplay(ctx, fmt.Sprintf("NotFound: %s", t.Title))
  		if err != nil {
  			return fmt.Errorf("failed to set task display: %w", err)
  		}
  		fmt.Printf("Could not find the release date for %s\n", t.Title)
  	}

  	return nil
  }
  ```
</CodeGroup>

With this fix, and after redeploying the runners with the updated `PrintMovieStats` implementation, you can retry the job:

<CodeGroup>
  ```python Python theme={"system"}
  job_client.retry(job)
  job_client.display(job)
  ```

  ```go Go theme={"system"}
  _, err := client.Jobs.Retry(ctx, job.ID)
  ```
</CodeGroup>

<Frame>
  <img alt="Job retried successfully" />

  <img alt="Job retried successfully" />
</Frame>

Now the task logs show:

```plaintext Logs theme={"system"}
Movie release date not found title="Tilebox - The Movie"
Movie release date fetched title="The Avengers" released="04 May 2012"
```

<Note>
  The logs confirm that only two tasks were executed, resuming from the point of failure instead of re-executing all tasks.
</Note>

The job was retried and succeeded. The two tasks that completed before the failure were not re-executed.


# Runners
Source: https://docs.tilebox.com/workflows/concepts/runners

Learn how Tilebox tasks are executed, selected for execution, versioned, and compare release runners and direct runners.

Runners are continuously running processes that listen for new tasks to execute. They claim queued tasks, execute them, and report task results back to Tilebox. You can start multiple runners in parallel to execute tasks concurrently or to provide different hardware and network access.

<Frame>
  <img alt="Runner architecture showing jobs submitted to Tilebox and a runner receiving assigned tasks, executing them, reporting results, and optionally submitting subtasks" />

  <img alt="Runner architecture showing jobs submitted to Tilebox and a runner receiving assigned tasks, executing them, reporting results, and optionally submitting subtasks" />
</Frame>

## Runner modes

Tilebox supports two runner modes. A **release runner** is started with the Tilebox CLI, loads [workflow releases](/workflows/concepts/workflow-releases) deployed to its cluster, and reacts to updated cluster deployments while it runs. A **direct runner** is a standalone script, service, or binary that uses the Tilebox SDK to connect to the API and register tasks directly.
Release runners still run in an environment you control, but the workflow code they execute is selected through cluster deployments. This separates compute operations from workflow release rollout. Direct runners are scaled and rolled out by your own infrastructure.

<Frame>
  <img alt="Release runner and direct runner modes both claim tasks from the Tilebox API after discovering task identifiers from deployments or SDK code" />

  <img alt="Release runner and direct runner modes both claim tasks from the Tilebox API after discovering task identifiers from deployments or SDK code" />
</Frame>

The two modes differ in how the runner gets its task registrations and how you roll out code changes.

|                  | Release runner                                                                                                            | Direct runner                                                         |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| Executable tasks | Loaded from workflow releases deployed to the runner's cluster                                                            | Registered directly in your script, service, or binary                |
| Runtime          | [Tilebox CLI](/agents-and-ai-tools/tilebox-cli.mdx) invokes the Python workflow project runtime from the release artifact | Your python script or Go binary, implemented with the Tilebox SDK     |
| Start command    | `tilebox runner start --cluster <cluster-slug>`                                                                           | `python runner.py`, `./my-runner-binary`, or your own deployment      |
| rollout model    | You publish releases and deploy them to clusters, the runner automatically picks up deployment changes                    | You deploy, restart, scale, and roll back the runner process yourself |
| Best for         | Reproducible releases, fast cluster deployments, and AI-assisted workflow iteration                                       | Custom deployments, Go runners, and direct SDK control                |

### Release runners

A release runner runs Python workflow releases deployed to a cluster. Start it with the Tilebox CLI:

```bash theme={"system"}
tilebox runner start --cluster dev-cluster
```

The release runner can run releases from multiple workflows at the same time, but only one release per workflow. It continuously polls the selected cluster for deployment updates, downloads missing release artifacts, checks them, starts Python processes for each workflow release, and requests work for every task identifier from its deployed releases. When a new release is deployed or removed, the runner updates the task set it can execute.

<Info>
  Release runners currently only support Python workflow projects. The Tilebox CLI invokes the Python runner environment from the published release artifact using `uv`.
</Info>

### Direct runners

A direct runner connects to the Tilebox API from your own code. Use it when you want full control over the process, deployment environment, dependencies, startup behavior, and scaling. You are responsible for deploying the script or binary, keeping it running, rolling out code changes, and rolling back when needed.

Define a `Runner` instance once and connect it to a `Client` during startup.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client, Runner
  from my_workflow.tasks import MyTask, OtherTask

  runner = Runner(tasks=[MyTask, OtherTask])

  if __name__ == "__main__":
      client = Client()
      runner.connect_to(client, cluster="dev-cluster").run_forever()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log/slog"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1/runner"
  	"github.com/my_org/myworkflow"
  )

  func main() {
  	ctx := context.Background()
  	client := workflows.NewClient()

  	workflowRunner, err := client.NewTaskRunner(ctx, runner.WithClusterSlug("dev-cluster"))
  	if err != nil {
  		slog.Error("failed to create runner", slog.Any("error", err))
  		return
  	}

  	if err := workflowRunner.RegisterTasks(&myworkflow.MyTask{}, &myworkflow.OtherTask{}); err != nil {
  		slog.Error("failed to register tasks", slog.Any("error", err))
  		return
  	}

  	workflowRunner.RunForever(ctx)
  }
  ```
</CodeGroup>

## Task selection

For a runner to pick up a submitted task, these conditions must match:

1. The task was submitted to the same [cluster](/workflows/concepts/clusters) as the runner.
2. The runner advertises a [task identifier with the same name](/workflows/concepts/tasks#task-identifiers) and a [compatible version](/workflows/concepts/tasks#semantic-versioning).
3. The task must be in `QUEUED` [state](/workflows/concepts/tasks#task-states), its [dependencies](/workflows/concepts/tasks#dependencies) are met, and its [maximum retries](/workflows/concepts/tasks#retry-handling) aren't exhausted.

Release runners advertise the task identifiers from workflow releases currently deployed to the cluster. Direct runners advertise the task identifiers they register in the running process.

<Tip>
  If multiple tasks match those conditions, Tilebox picks one and assigns it to a runner. The remaining tasks stay queued until another matching runner is available. Parallel runner processes can speed up the job execution in such cases.
</Tip>

## Parallelism

Start multiple runner processes to execute tasks in parallel. Each runner process claims and executes tasks independently. You can run multiple release runners, multiple direct runners, or a mix of both in the same cluster. This increases parallelism and helps handle large workloads.

To test this, run multiple instances of the runner script in different terminal windows on your local machine, or use the [CLI](/agents-and-ai-tools/tilebox-cli) built-in `parallel` subcommand to start multiple runners in parallel.

```bash theme={"system"}
# start multiple release runners in parallel
> tilebox parallel -n 5 -- tilebox runner start --cluster <dev-cluster>

# or direct runner mode
> tilebox parallel -n 5 -- python your_direct_runner.py
```

## Scaling

One key benefit of this runner architecture is the **ability to scale even while workflows are executing**. You can start new runners at any time, and they can immediately pick up queued tasks to execute. You do not need an entire processing cluster available at the start of a workflow, because you can start and stop more runners as needed.

This is particularly beneficial in cloud environments, where runners can be automatically started and stopped based on current workload, measured by metrics such as CPU usage. Here's an example scenario:

1. A single runner process is actively waiting for work in a cloud environment.
2. A large workload is submitted to the workflow orchestrator, resulting in the runner picking up the first task.
3. The first task creates new sub-tasks for processing, which the runner also picks up.
4. As the workload increases, the runner's CPU usage rises, triggering the cloud environment to automatically start up new runner instances.
5. Newly started runners begin executing queued tasks, distributing the workload among all available runners.
6. Once the workload decreases, the cloud environment automatically stops some runners.
7. The remaining work continues while runner instances are scaled back down, until everything is done.
8. Only a single runner remains idle until new tasks arrive.

CPU usage-based auto scaling is just one method to scale runners. Other metrics, such as memory usage or network bandwidth, are also supported by many cloud environments.

<Note>In a future release, configuration options for scaling runners based on custom metrics (for example the number of queued tasks) are planned.</Note>

## Distributed Execution

Runners can be distributed across different compute environments. For instance, some data stored on-premise may need pre-processing, while further processing occurs in the cloud. A job might involve tasks that filter relevant on-premise data and publish it to the cloud, and other tasks that read data from the cloud and process it. In such scenarios, one runner can run on-premise and another in a cloud environment, resulting in them effectively collaborating on the same job.

Another advantage of distributed runners is executing workflows that require specific hardware for certain tasks. For example, one task might need a GPU, while another requires extensive memory.

Here's an example of a distributed workflow:

<CodeGroup title="Distributed Workflow">
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class DistributedWorkflow(Task):
      def execute(self, context: ExecutionContext) -> None:
          download_task = context.submit_subtask(DownloadData())
          process_task = context.submit_subtask(
            ProcessData(),
            depends_on=[download_task],
          )

  class DownloadData(Task):
      """
      Download a dataset and store it in a shared internal bucket.
      Requires a good network connection for high download bandwidth.
      """
      def execute(self, context: ExecutionContext) -> None:
          pass

  class ProcessData(Task):
      """
      Perform compute-intensive processing of a dataset.
      The dataset must be available in an internal bucket.
      Requires access to a GPU for optimal performance.
      """
      def execute(self, context: ExecutionContext) -> None:
          pass
  ```

  ```go Go theme={"system"}
  package distributed

  import (
  	"context"
  	"fmt"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1/subtask"
  )

  type DistributedWorkflow struct{}

  func (t *DistributedWorkflow) Execute(ctx context.Context) error {
  	downloadTask, err := workflows.SubmitSubtask(ctx, &DownloadData{})
  	if err != nil {
  		return fmt.Errorf("failed to submit download subtask: %w", err)
  	}

  	_, err = workflows.SubmitSubtask(ctx, &ProcessData{}, subtask.WithDependencies(downloadTask))
  	if err != nil {
  		return fmt.Errorf("failed to submit process subtask: %w", err)
  	}
  	return nil
  }

  // DownloadData Download a dataset and store it in a shared internal bucket.
  // Requires a good network connection for high download bandwidth.
  type DownloadData struct{}

  func (t *DownloadData) Execute(ctx context.Context) error {
  	return nil
  }

  // ProcessData Perform compute-intensive processing of a dataset.
  // The dataset must be available in an internal bucket.
  // Requires access to a GPU for optimal performance.
  type ProcessData struct{}

  func (t *ProcessData) Execute(ctx context.Context) error {
  	return nil
  }
  ```
</CodeGroup>

To achieve distributed execution for this workflow, no single runner capable of executing all three of the tasks is set up.
Instead, two runners, each capable of executing one of the tasks, are set up: one in a high-speed network environment and the other with GPU access.
When the distributed workflow runs, the first runner picks up the `DownloadData` task, while the second picks up the `ProcessData` task.
The `DistributedWorkflow` does not require specific hardware, so it can be registered with both runners and executed by either one.

<Tabs>
  <Tab title="download_runner.py">
    <CodeGroup title="Distributed Runners">
      ```python Python theme={"system"}
      from tilebox.workflows import Client

      client = Client()
      high_network_speed_runner = client.runner(
          tasks=[DownloadData, DistributedWorkflow]
      )
      high_network_speed_runner.run_forever()
      ```

      ```go Go theme={"system"}
      package main

      import (
      	"context"
      	"log/slog"

      	"github.com/tilebox/tilebox-go/workflows/v1"
      )

      func main() {
      	ctx := context.Background()
      	client := workflows.NewClient()

      	highNetworkSpeedRunner, err := client.NewTaskRunner(ctx)
      	if err != nil {
      		slog.Error("failed to create runner", slog.Any("error", err))
      		return
      	}

      	err = highNetworkSpeedRunner.RegisterTasks(
      		&DownloadData{},
      		&DistributedWorkflow{},
      	)
      	if err != nil {
      		slog.Error("failed to register tasks", slog.Any("error", err))
      		return
      	}

      	highNetworkSpeedRunner.RunForever(ctx)
      }
      ```
    </CodeGroup>
  </Tab>

  <Tab title="gpu_runner.py">
    <CodeGroup title="Distributed Runners">
      ```python Python theme={"system"}
      from tilebox.workflows import Client

      client = Client()
      gpu_runner = client.runner(
          tasks=[ProcessData, DistributedWorkflow]
      )
      gpu_runner.run_forever()
      ```

      ```go Go theme={"system"}
      package main

      import (
      	"context"
      	"log/slog"

      	"github.com/tilebox/tilebox-go/workflows/v1"
      )

      func main() {
      	ctx := context.Background()
      	client := workflows.NewClient()

      	gpuRunner, err := client.NewTaskRunner(ctx)
      	if err != nil {
      		slog.Error("failed to create runner", slog.Any("error", err))
      		return
      	}

      	err = gpuRunner.RegisterTasks(
      		&ProcessData{},
      		&DistributedWorkflow{},
      	)
      	if err != nil {
      		slog.Error("failed to register tasks", slog.Any("error", err))
      		return
      	}

      	gpuRunner.RunForever(ctx)
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

Now, both `download_runner.py` and `gpu_runner.py` are started, in parallel, on different machines with the required hardware for each. When `DistributedWorkflow` is submitted, it executes on one of the two runners, and it's submitted sub-tasks are handled by the appropriate runner.

In this case, since `ProcessData` depends on `DownloadData`, the GPU runner remains idle until the download completion, then picks up the processing task.

<Note>
  You can also differentiate between runners by specifying different [clusters](/workflows/concepts/clusters) and choosing specific clusters for sub-task submissions. For more details, see the [Clusters](/workflows/concepts/clusters) section.
</Note>

## Task Failures

If an unhandled exception occurs during task execution, the runner captures it and reports it back to the workflow orchestrator. The orchestrator then marks the task as failed, leading to [job cancellation](/workflows/concepts/jobs#cancellation) to prevent further tasks of the same job-that may not be relevant anymore-from being executed.

A task failure does not result in losing all previous work done by the job. If the failure is fixable—by fixing a bug in a task implementation, ensuring the task has necessary resources, or simply retrying it due to a flaky network connection—it may be worth [retrying](/workflows/concepts/jobs#retries) the job.

When retrying a job, all failed tasks are added back to the queue, allowing a runner to potentially execute them. If execution then succeeds, the job continues smoothly. Otherwise, the task will remain marked as failed and can be retried again if desired.

For a release runner, publish a compatible fixed release and deploy it to the same cluster before retrying. For a direct runner, deploy the fixed script or binary before retrying. Keep task identifiers and input schemas compatible when you want an existing failed job to resume from the point of failure.

## Task idempotency

Since a task may be retried, it's possible that a task is executed more than once. Depending on where in the execution of the task it failed, it may have already performed some side effects, such as writing to a database, or sending a message to a queue. Because of that it's crucial to ensure that tasks are [idempotent](https://en.wikipedia.org/wiki/Idempotence). Idempotent tasks can be executed multiple times without altering the outcome beyond the first successful execution.

A special case of idempotency involves submitting sub-tasks. After a task calls `context.submit_subtask` and then fails and is retried, those submitted sub-tasks of an earlier failed execution are automatically removed, ensuring that they can be safely submitted again when the task is retried.

## Runner Crashes

Tilebox Workflows has an internal mechanism to handle unexpected runner crashes. When a runner picks up a task, it periodically sends a heartbeat to the workflow orchestrator. If the orchestrator does not receive this heartbeat for a defined duration, it marks the task as failed and automatically attempts to [retry](/workflows/concepts/jobs#retries) it up to 10 times. This allows another runner to pick up the task and continue executing the job.

This mechanism ensures that scenarios such as power outages, hardware failures, or dropped network connections are handled effectively, preventing any task from remaining in a running state indefinitely.

## Observability

Tilebox captures logs, spans, task states, and runner context from both runner modes. Use [Workflow observability](/workflows/run-and-inspect/introduction) to inspect job execution, task failures, and runner behavior.


# Understanding and Creating Tasks
Source: https://docs.tilebox.com/workflows/concepts/tasks

Define workflow tasks, inputs, subtasks, dependencies, retries, and stable task identifiers.

A task is the unit of work Tilebox [runners](/workflows/concepts/runners) execute. A task class defines the code to run, the input fields that are serialized with each task submission, and optional relationships to other tasks through subtasks and dependencies.

Tasks can run as the root task of a [job](/workflows/concepts/jobs) or as subtasks submitted by another task. This lets a workflow build a dynamic task graph while Tilebox schedules eligible tasks across runners in the selected [cluster](/workflows/concepts/clusters).

## Creating a Task

To create a task in Tilebox, define a class that extends the `Task` base class and implements the `execute` method. The `execute` method is the entry point for the task where its logic is defined. It's called when the task is executed.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class MyFirstTask(Task):
      def execute(self, context: ExecutionContext):
          print("Hello World!")
  ```

  ```go Go theme={"system"}
  type MyFirstTask struct{}

  func (t *MyFirstTask) Execute(ctx context.Context) error {
  	slog.Info("Hello World!")
  	return nil
  }
  ```
</CodeGroup>

This example demonstrates a simple task that prints "Hello World!" to the console.

For python, the key components of this task are:

<AccordionGroup>
  <Accordion title="class MyFirstTask(Task)">
    `MyFirstTask` is a subclass of the `Task` class, which serves as the base class for all defined tasks. It provides the essential structure for a task. Inheriting from `Task` automatically makes the class a `dataclass`, which is useful [for specifying inputs](#input-parameters). Additionally, by inheriting from `Task`, the task is automatically assigned an [identifier based on the class name](#task-identifiers).
  </Accordion>

  <Accordion title="def execute">
    The `execute` method is the entry point for executing the task. This is where the task's logic is defined. It's invoked by a [runner](/workflows/concepts/runners) when the task runs and performs the task's operation.
  </Accordion>

  <Accordion title="context: ExecutionContext">
    The `context` argument is an `ExecutionContext` instance that provides access to an [API for submitting new tasks](/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask) as part of the same job, [task logging](/api-reference/python/tilebox.workflows/ExecutionContext.logger), [custom tracing](/api-reference/python/tilebox.workflows/ExecutionContext.tracer), and features like [shared caching](/api-reference/python/tilebox.workflows/ExecutionContext.job_cache).
  </Accordion>
</AccordionGroup>

For Go, the key components are:

<AccordionGroup>
  <Accordion title="type MyFirstTask struct{}">
    `MyFirstTask` is a struct that implements the `Task` interface. It represents the task to be executed.
  </Accordion>

  <Accordion title="func (t *MyFirstTask) Execute(ctx context.Context) error">
    The `Execute` method is the entry point for executing the task. This is where the task's logic is defined. It's invoked by a [runner](/workflows/concepts/runners) when the task runs and performs the task's operation.
  </Accordion>
</AccordionGroup>

<Note>
  The code samples on this page do not illustrate how to execute the task. That will be covered in the
  [next section on runners](/workflows/concepts/runners). The reason for that is that executing tasks is a separate concern from implementing tasks.
</Note>

## Input Parameters

Tasks often require input parameters to operate. These inputs can range from simple values to complex data structures. By inheriting from the `Task` class, the task is treated as a Python `dataclass`, allowing input parameters to be defined as class attributes.

<Info>
  Tasks must be **serializable to JSON or to protobuf** because they may be distributed across a cluster of [runners](/workflows/concepts/runners).
</Info>

<Info>
  In Go, task parameters must be exported fields of the task struct (starting with an uppercase letter), otherwise they will not be serialized to JSON.
</Info>

Supported types for input parameters include:

* Basic types such as `str`, `int`, `float`, `bool`
* Lists and dictionaries of basic types
* Nested data classes that are also JSON-serializable or protobuf-serializable

<CodeGroup>
  ```python Python theme={"system"}
  class ParametrizableTask(Task):
      message: str
      number: int
      data: dict[str, str]

      def execute(self, context: ExecutionContext):
          print(self.message * self.number)

  task = ParametrizableTask("Hello", 3, {"key": "value"})
  ```

  ```go Go theme={"system"}
  type ParametrizableTask struct {
    Message string
    Number  int
    Data    map[string]string
  }

  func (t *ParametrizableTask) Execute(context.Context) error {
    slog.Info(strings.Repeat(t.Message, t.Number))
    return nil
  }

  task := &ParametrizableTask{
    message: "Hello",
    number:  3,
    data:    map[string]string{"key": "value"},
  }
  ```
</CodeGroup>

## Task Composition and subtasks

Until now, tasks have performed only a single operation. But tasks can be more powerful. **Tasks can submit other tasks as subtasks.** This allows for a modular workflow design, breaking down complex operations into simpler, manageable parts. Additionally, the execution of subtasks is automatically parallelized whenever possible.

<CodeGroup>
  ```python Python theme={"system"}
  class ParentTask(Task):
      num_subtasks: int

      def execute(self, context: ExecutionContext) -> None:
          for i in range(self.num_subtasks):
              context.submit_subtask(ChildTask(i))

  class ChildTask(Task):
      index: int

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Executing child task", index=self.index)

  # after submitting this task, a runner may pick it up and execute it
  # which will result in 5 ChildTasks being submitted and executed as well
  task = ParentTask(5)
  ```

  ```go Go theme={"system"}
  type ParentTask struct {
  	NumSubtasks int
  }

  func (t *ParentTask) Execute(ctx context.Context) error {
  	for i := range t.NumSubtasks {
  		_, err := workflows.SubmitSubtask(ctx, &ChildTask{Index: i})
  		if err != nil {
  			return err
  		}
  	}

  	return nil
  }

  type ChildTask struct {
  	Index int
  }

  func (t *ChildTask) Execute(context.Context) error {
  	slog.Info("Executing ChildTask", slog.Int("index", t.Index))
  	
  	return nil
  }

  // after submitting this task, a runner may pick it up and execute it 
  // which will result in 5 ChildTasks being submitted and executed as well
  task := &ParentTask{numSubtasks: 5}
  ```
</CodeGroup>

In this example, a `ParentTask` submits `ChildTask` tasks as subtasks. The number of subtasks to be submitted is based on the `num_subtasks` attribute of the `ParentTask`. The `submit_subtask` method takes an instance of a task as its argument, meaning the task to be submitted must be instantiated with concrete parameters first.

Parent task do not have access to results of subtasks, instead, tasks can use [shared caching](/workflows/run-and-inspect/caches#storing-and-retrieving-data) to share data between tasks.

<Info>
  By submitting a task as a subtask, its execution is scheduled as part of the same job as the parent task. Compared to just directly invoking the subtask's `execute` method, this allows the subtask's execution to occur on a different machine or in parallel with other subtasks. To learn more about how tasks are executed, see the section on [runners](/workflows/concepts/runners).
</Info>

### Larger subtasks example

A practical workflow example showcasing task composition might help illustrate the capabilities of tasks. Below is an example of a set of tasks forming a workflow capable of downloading a set number of random dog images from the internet. The [Dog API](https://thedogapi.com/) can be used to get the image URLs, and then download them. Implementing this using Task Composition could look like this:

<CodeGroup title="Task Composition">
  ```python Python theme={"system"}
  import httpx  # pip install httpx
  from pathlib import Path

  class DownloadRandomDogImages(Task):
      num_images: int

      def execute(self, context: ExecutionContext) -> None:
          url = f"https://api.thedogapi.com/v1/images/search?limit={self.num_images}"
          response = httpx.get(url)
          for dog_image in response.json():
              context.submit_subtask(DownloadImage(dog_image["url"]))

  class DownloadImage(Task):
      url: str

      def execute(self, context: ExecutionContext) -> None:
          file = Path("dogs") / self.url.split("/")[-1]
          response = httpx.get(self.url)
          with file.open("wb") as file:
              file.write(response.content)
  ```

  ```go Go theme={"system"}
  package dogs

  import (
  	"context"
  	"encoding/json"
  	"fmt"
  	"io"
  	"net/http"
  	"os"
  	"strings"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type DogImage struct {
  	ID     string `json:"id"`
  	URL    string `json:"url"`
  	Width  *int   `json:"width"`
  	Height *int   `json:"height"`
  }

  type DownloadRandomDogImages struct {
  	NumImages int
  }

  func (t *DownloadRandomDogImages) Execute(ctx context.Context) error {
  	url := fmt.Sprintf("https://api.thedogapi.com/v1/images/search?limit=%d", t.NumImages)
  	response, err := http.Get(url)
  	if err != nil {
  		return fmt.Errorf("failed to download images: %w", err)
  	}

  	defer response.Body.Close()
  	body, err := io.ReadAll(response.Body)
  	if err != nil {
  		return fmt.Errorf("failed to read response: %w", err)
  	}

  	var dogImages []DogImage
  	err = json.Unmarshal(body, &dogImages)
  	if err != nil {
  		return err
  	}

  	for _, dogImage := range dogImages {
  		_, err := workflows.SubmitSubtask(ctx, &DownloadImage{URL: dogImage.URL})
  		if err != nil {
  			return err
  		}
  	}
  	return nil
  }

  type DownloadImage struct {
  	URL string
  }

  func (t *DownloadImage) Execute(context.Context) error {
  	response, err := http.Get(t.URL)
  	if err != nil {
  		return fmt.Errorf("failed to download image: %w", err)
  	}

  	defer response.Body.Close()
  	body, err := io.ReadAll(response.Body)
  	if err != nil {
  		return fmt.Errorf("failed to read response: %w", err)
  	}

  	err = os.MkdirAll("dogs", 0o755)
  	if err != nil {
  		return fmt.Errorf("failed to create dogs directory: %w", err)
  	}

  	elements := strings.Split(t.URL, "/")
  	file := fmt.Sprintf("dogs/%s", elements[len(elements)-1])

  	return os.WriteFile(file, body, 0o600)
  }
  ```
</CodeGroup>

This example consists of the following tasks:

<AccordionGroup>
  <Accordion title="DownloadRandomDogImages">
    `DownloadRandomDogImages` fetches a specific number of random dog image URLs from an API. It then submits a `DownloadImage` task for each received image URL.
  </Accordion>

  <Accordion title="DownloadImage">
    `DownloadImage` downloads an image from a specified URL and saves it to a file.
  </Accordion>
</AccordionGroup>

Together, these tasks create a workflow that downloads random dog images from the internet. The relationship between the two tasks and their formation as a workflow becomes clear when `DownloadRandomDogImages` submits `DownloadImage` tasks as subtasks.

Visualizing the execution of such a workflow is akin to a tree structure where the `DownloadRandomDogImages` task is the root, and the `DownloadImage` tasks are the leaves. For instance, when downloading five random dog images, the following tasks are executed.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  jobs = client.jobs()
  job = jobs.submit(
      "download-dog-images",
      DownloadRandomDogImages(5),
  )

  # now our deployed runners will pick up the task and execute it

  jobs.display(job)
  ```

  ```go Go theme={"system"}
  ctx := context.Background()
  client := workflows.NewClient()

  job, err := client.Jobs.Submit(ctx, "download-dog-images",
    []workflows.Task{
  	&helloworld.DownloadRandomDogImages{
  		NumImages: 5,
  	},
    },
  )
  if err != nil {
    slog.Error("Failed to submit job", slog.Any("error", err))
    return
  }

  // now our deployed runners will pick up the task and execute it
  ```
</CodeGroup>

<Frame>
  <img alt="Download Dog Images Workflow" />

  <img alt="Download Dog Images Workflow" />
</Frame>

In total, six tasks are executed: the `DownloadRandomDogImages` task and five `DownloadImage` tasks. The `DownloadImage` tasks can execute in parallel, as they are independent. If more than one runner is available, the Tilebox Workflow Orchestrator **automatically parallelizes** the execution of these tasks.

<Tip>
  Check out [job\_client.display](/workflows/concepts/jobs#visualization) to learn how this visualization was automatically generated from the task executions.
</Tip>

## Task States

Every task goes through a set of states during its lifetime.

* When submitted, either as a job or as a subtask, it starts in the `QUEUED` state and transitions to `RUNNING` when a runner picks it up.
* If the task executes successfully, it transitions to `COMPUTED`.
* If the task fails, it transitions to `FAILED`, unless it's an [optional task](#optional-tasks), or nested within an [optional task](#nested-optional-tasks), in which case it transitions to `FAILED_OPTIONAL`.
* As soon as all subtasks of a task are `COMPUTED` (or `FAILED_OPTIONAL`), the task is considered `COMPLETED`, allowing dependent tasks to be executed.

The table below summarizes the different task states and their meanings.

| Task State            | Description                                                                                                                                                                                                                         |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Queued**            | The task is queued and waiting for execution. Any [eligible](/workflows/concepts/runners#task-selection) runner can pick it up and execute it, as soon as it's parent task is `COMPUTED` and all it's dependencies are `COMPLETED`. |
| **Running**           | The task is currently being executed by a runner.                                                                                                                                                                                   |
| **Computed**          | The task has successfully been computed, but still has outstanding subtasks.                                                                                                                                                        |
| **Completed**         | The task has successfully been computed, and all it's subtasks are also computed, making it `COMPLETED`. This is the final state of a task. Only once a task has been `COMPLETED`, dependent tasks can be executed.                 |
| **Failed**            | The task has been executed but encountered an error.                                                                                                                                                                                |
| **Failed (Optional)** | The task has been executed but encountered an error. Since the task was [marked as optional](#optional-tasks), the job continues executing.                                                                                         |
| **Skipped**           | The task was skipped because it's a subtask of an optional task and one of its siblings failed.                                                                                                                                     |

<Frame>
  <img alt="Task States" />

  <img alt="Task States" />
</Frame>

## Map-Reduce Pattern

Often times the input to a task is a list, with elements that should then be **mapped** to individual subtasks, whose results are later aggregated in a **reduce** step. This pattern is commonly known as [MapReduce](https://en.wikipedia.org/wiki/MapReduce) and a common pattern in workflows. In Tilebox, the reduce step is typically defined as a separate task that depends on all the map tasks.

For example, the workflow below applies this pattern to a list of numbers to calculate the sum of all squares of the numbers. The `Square` task takes a single number and squares it, and the `Sum` task reduces the list of squared numbers to a single sum.

<CodeGroup title="Map-Reduce">
  ```python Python theme={"system"}
  class SumOfSquares(Task):
      numbers: list[int]

      def execute(self, context: ExecutionContext) -> None:
  		# 1. Map
          square_tasks = context.submit_subtasks(
              [Square(num) for num in self.numbers]
          )
          # 2. Reduce
          sum_task = context.submit_subtask(Sum(), depends_on=square_tasks)


  class Square(Task):  # The map step
      num: int

      def execute(self, context: ExecutionContext) -> None:
          result = self.num ** 2
          # typically the output of a task is a large dataset,
          # so we save individual results into a shared cache
          context.job_cache.group("squares")[str(self.num)] = str(result).encode()
          context.current_task.display = f"Square({self.num})"

  class Sum(Task):  # The reduce step
      def execute(self, context: ExecutionContext) -> None:
          result = 0
          # access our cached results from the map step
          squares = context.job_cache.group("squares")
          for key in squares:
              result += int(squares[key].decode())

          context.logger.info("Computed sum of squares", result=result)
  ```
</CodeGroup>

Submitting a job of the `SumOfSquares` task and running it with a runner can be done as follows:

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import InMemoryCache

  client = Client()
  jobs = client.jobs()
  job = jobs.submit(
      "sum-of-squares",
      SumOfSquares([12, 345, 453, 21, 45, 98]),
  )

  client.runner(tasks=[SumOfSquares, Square, Sum], cache=InMemoryCache()).run_all()

  jobs.display(job)
  ```
</CodeGroup>

```plaintext Logs theme={"system"}
Computed sum of squares result=336448
```

<Frame>
  <img alt="Sum of squares workflow using the map-reduce pattern" />

  <img alt="Sum of squares workflow using the map-reduce pattern" />
</Frame>

## Recursive subtasks

Tasks can not only submit other tasks as subtasks, but also instances of themselves. This allows for a recursive breakdown of a task into smaller chunks. Such recursive decomposition algorithms are referred to as [divide and conquer algorithms](https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm).
For example, the `RecursiveTask` below is a valid task that submits smaller instances of itself as subtasks.

<Warning>When implementing a recursive task, it's important to define a base case that stops the recursion. Otherwise, the task will keep submitting subtasks indefinitely, resulting in an infinite loop.</Warning>

<CodeGroup title="Recursive Subtasks">
  ```python Python theme={"system"}
  class RecursiveTask(Task):
      num: int

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Executing recursive task", num=self.num)
  		# if num < 2, we reached the base case and stop recursion
          if self.num >= 2:
              context.submit_subtask(RecursiveTask(self.num // 2))
  ```

  ```go Go theme={"system"}
  type RecursiveTask struct {
    Num int
  }

  func (t *RecursiveTask) Execute(ctx context.Context) error {
    slog.Info("Executing RecursiveTask", slog.Int("num", t.Num))
    // if num < 2, we reached the base case and stop recursion
    if t.Num >= 2 {
      _, err := workflows.SubmitSubtask(ctx, &RecursiveTask{Num: t.Num / 2})
      if err != nil {
        return err
      }
    }
    return nil
  }
  ```
</CodeGroup>

### Recursive subtask example

An example for this is the [random dog images workflow](#larger-subtasks-example) mentioned earlier. In the previous implementation, downloading images was already parallelized. But the initial orchestration of the individual download tasks was not parallelized, because `DownloadRandomDogImages` was responsible for fetching all random dog image URLs and only submitted the individual download tasks once all URLs were retrieved. For a large number of images this setup can bottleneck the entire workflow.

To improve this, recursive subtask submission decomposes a `DownloadRandomDogImages` task with a high number of images into two smaller `DownloadRandomDogImages` tasks, each fetching half. This process can be repeated until a specified threshold is met, at which point the Dog API can be queried directly for image URLs. That way, image downloads start as soon as the first URLs are retrieved, without initial waiting.

An implementation of this recursive submission may look like this:

<CodeGroup title="Task Composition">
  ```python Python theme={"system"}
  class DownloadRandomDogImages(Task):
      num_images: int

      def execute(self, context: ExecutionContext) -> None:
          if self.num_images > 4:
              half = self.num_images // 2
              remaining = self.num_images - half  # account for odd numbers
              context.submit_subtask(DownloadRandomDogImages(half))
              context.submit_subtask(DownloadRandomDogImages(remaining))
          else:
              url = f"https://api.thedogapi.com/v1/images/search?limit={self.num_images}"
              response = httpx.get(url)
              for dog_image in response.json()[:self.num_images]:
                  context.submit_subtask(DownloadImage(dog_image["url"]))
  ```

  ```go Go theme={"system"}
  type DownloadRandomDogImages struct {
  	NumImages int
  }

  func (t *DownloadRandomDogImages) Execute(ctx context.Context) error {
  	if t.NumImages > 4 {
  		half := t.NumImages / 2
  		remaining := t.NumImages - half // account for odd numbers
  		_, err := workflows.SubmitSubtask(ctx, &DownloadRandomDogImages{NumImages: half})
  		if err != nil {
  			return err
  		}
  		_, err = workflows.SubmitSubtask(ctx, &DownloadRandomDogImages{NumImages: remaining})
  		if err != nil {
  			return err
  		}
  	} else {
  		url := fmt.Sprintf("https://api.thedogapi.com/v1/images/search?limit=%d", t.NumImages)
  		response, err := http.Get(url)
  		if err != nil {
  			return fmt.Errorf("failed to download images: %w", err)
  		}

  		defer response.Body.Close()
  		body, err := io.ReadAll(response.Body)
  		if err != nil {
  			return fmt.Errorf("failed to read response: %w", err)
  		}

  		var dogImages []DogImage
  		err = json.Unmarshal(body, &dogImages)
  		if err != nil {
  			return err
  		}

  		for _, dogImage := range dogImages {
  			_, err := workflows.SubmitSubtask(ctx, &DownloadImage{URL: dogImage.URL})
  			if err != nil {
  				return err
  			}
  		}
  	}
  	return nil
  }
  ```
</CodeGroup>

With this implementation, downloading a large number of images (for example, 9) results in the following tasks being executed:

<Frame>
  <img alt="Download Dog Images Workflow implemented recursively" />

  <img alt="Download Dog Images Workflow implemented recursively" />
</Frame>

## Retry Handling

By default, when a task fails to execute, it's marked as failed. In some cases, it may be useful to retry the task multiple times before marking it as a failure. This is particularly useful for tasks dependent on external services that might be temporarily unavailable.

Tilebox Workflows allows you to specify the number of retries for a task using the `max_retries` argument of the `submit_subtask` method.

Check out the example below to see how this might look like in practice.

<Note>
  A failed task may be picked up by any available runner and not necessarily the same one that it failed on.
</Note>

<CodeGroup title="Submitting Subtasks">
  ```python Python theme={"system"}
  import random

  class RootTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.submit_subtask(FlakyTask(), max_retries=5)

  class FlakyTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Executing flaky task")

          if random.random() < 0.1:
              raise Exception("FlakyTask failed randomly")
  ```

  ```go Go theme={"system"}
  package flaky

  import (
  	"context"
  	"errors"
  	"log/slog"
  	"math/rand/v2"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1/subtask"
  )

  type RootTask struct{}

  func (t *RootTask) Execute(ctx context.Context) error {
  	_, err := workflows.SubmitSubtask(ctx, &FlakyTask{},
  		subtask.WithMaxRetries(5),
  	)
  	return err
  }

  type FlakyTask struct{}

  func (t *FlakyTask) Execute(context.Context) error {
  	slog.Info("Executing FlakyTask")

  	if rand.Float64() < 0.1 {
  		return errors.New("FlakyTask failed randomly")
  	}
  	return nil
  }
  ```
</CodeGroup>

## Dependencies

Tasks often rely on other tasks. For example, a task that processes data might depend on a task that fetches that data. **Tasks can express their dependencies on other tasks** by using the `depends_on` argument of the [`submit_subtask`](/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask) method. This means that a dependent task will only execute after the task it relies on has successfully completed.

<Note>
  The `depends_on` argument accepts a list of tasks, enabling a task to depend on multiple other tasks.
</Note>

A workflow with dependencies might look like this:

<CodeGroup title="Task Composition">
  ```python Python theme={"system"}
  class RootTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          first_task = context.submit_subtask(
            PrintTask("Executing first")
          )
          second_task = context.submit_subtask(
            PrintTask("Executing second"), 
            depends_on=[first_task],
          )
          third_task = context.submit_subtask(
            PrintTask("Executing last"),
            depends_on=[second_task],
          )

  class PrintTask(Task):
      message: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Print task executed", message=self.message)
  ```

  ```go Go theme={"system"}
  type RootTask struct{}

  func (t *RootTask) Execute(ctx context.Context) error {
  	firstTask, err := workflows.SubmitSubtask(
  		ctx,
  		&PrintTask{Message: "Executing first"},
  	)
  	if err != nil {
  		return err
  	}

  	secondTask, err := workflows.SubmitSubtask(
  		ctx,
  		&PrintTask{Message: "Executing second"},
  		subtask.WithDependencies(firstTask),
  	)
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(
  		ctx,
  		&PrintTask{Message: "Executing last"},
  		subtask.WithDependencies(secondTask),
  	)
  	if err != nil {
  		return err
  	}

  	return nil
  }

  type PrintTask struct {
  	Message string
  }

  func (t *PrintTask) Execute(context.Context) error {
  	slog.Info("PrintTask", slog.String("message", t.Message))
  	return nil
  }
  ```
</CodeGroup>

The `RootTask` submits three `PrintTask` tasks as subtasks. These tasks depend on each other, meaning the second task executes only after the first task has successfully completed, and the third only executes after the second completes. The tasks are executed sequentially.

<Note>
  If a task upon which another task depends submits subtasks, those subtasks must also execute before the dependent task begins execution.
</Note>

### Dependencies Example

A practical example is a workflow that fetches news articles from an API and processes them using the [News API](https://newsapi.org/).

<CodeGroup title="Task Dependencies">
  ```python Python theme={"system"}
  from pathlib import Path
  import json
  from collections import Counter
  import httpx  # pip install httpx

  class NewsWorkflow(Task):
      category: str
      max_articles: int

      def execute(self, context: ExecutionContext) -> None:
          fetch_task = context.submit_subtask(FetchNews(self.category, self.max_articles))
          context.submit_subtask(PrintHeadlines(), depends_on=[fetch_task])
          context.submit_subtask(MostFrequentAuthors(), depends_on=[fetch_task])

  class FetchNews(Task):
      category: str
      max_articles: int

      def execute(self, context: ExecutionContext) -> None:
          url = f"https://newsapi.org/v2/top-headlines?category={self.category}&pageSize={self.max_articles}&country=us&apiKey=API_KEY"
          with context.tracer.span("fetch-news") as span:
              span.set_attribute("category", self.category)
              span.set_attribute("max_articles", self.max_articles)
              news = httpx.get(url).json()
          # check out our documentation page on caches to learn
          # about a better way of passing data between tasks
          Path("news.json").write_text(json.dumps(news))
          context.logger.info(
              "Fetched news articles",
              category=self.category,
              article_count=len(news["articles"]),
          )

  class PrintHeadlines(Task):
      def execute(self, context: ExecutionContext) -> None:
          news = json.loads(Path("news.json").read_text())
          for article in news["articles"]:
              context.logger.info(
                  "News headline",
                  published_at=article["publishedAt"][:10],
                  title=article["title"],
              )

  class MostFrequentAuthors(Task):
      def execute(self, context: ExecutionContext) -> None:
          news = json.loads(Path("news.json").read_text())
          authors = [article["author"] for article in news["articles"]]
          for author, count in Counter(authors).most_common():
              context.logger.info("Author article count", author=author, count=count)

  # now submit a job, and then visualize it
  job = job_client.submit("process-news",
      NewsWorkflow(category="science", max_articles=5),
  )
  ```

  ```go Go theme={"system"}
  package news

  import (
  	"context"
  	"encoding/json"
  	"fmt"
  	"io"
  	"log/slog"
  	"net/http"
  	"os"
  	"time"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1/subtask"
  )

  const newsAPIKey = "YOUR_API_KEY"

  type NewsWorkflow struct {
  	Category    string
  	MaxArticles int
  }

  func (t *NewsWorkflow) Execute(ctx context.Context) error {
  	fetchTask, err := workflows.SubmitSubtask(ctx, &FetchNews{
  		Category:    t.Category,
  		MaxArticles: t.MaxArticles,
  	})
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(ctx, &PrintHeadlines{}, subtask.WithDependencies(fetchTask))
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(ctx, &MostFrequentAuthors{}, subtask.WithDependencies(fetchTask))
  	if err != nil {
  		return err
  	}

  	return nil
  }

  type News struct {
  	Status       string `json:"status"`
  	TotalResults int    `json:"totalResults"`
  	Articles     []struct {
  		Source struct {
  			ID   *string `json:"id"`
  			Name string  `json:"name"`
  		} `json:"source"`
  		Author      *string   `json:"author"`
  		Title       string    `json:"title"`
  		Description *string   `json:"description"`
  		URL         string    `json:"url"`
  		URLToImage  *string   `json:"urlToImage"`
  		PublishedAt time.Time `json:"publishedAt"`
  		Content     *string   `json:"content"`
  	} `json:"articles"`
  }

  type FetchNews struct {
  	Category    string
  	MaxArticles int
  }

  func (t *FetchNews) Execute(context.Context) error {
  	url := fmt.Sprintf("https://newsapi.org/v2/top-headlines?category=%s&pageSize=%d&country=us&apiKey=%s", t.Category, t.MaxArticles, newsAPIKey)
  	response, err := http.Get(url)
  	if err != nil {
  		return fmt.Errorf("failed to download news: %w", err)
  	}

  	defer response.Body.Close()
  	body, err := io.ReadAll(response.Body)
  	if err != nil {
  		return fmt.Errorf("failed to read response: %w", err)
  	}

  	// check out our documentation page on caches to learn
  	// about a better way of passing data between tasks
  	return os.WriteFile("news.json", body, 0o600)
  }

  type PrintHeadlines struct{}

  func (t *PrintHeadlines) Execute(context.Context) error {
  	newsBytes, err := os.ReadFile("news.json")
  	if err != nil {
  		return fmt.Errorf("failed to read news: %w", err)
  	}

  	var news News
  	err = json.Unmarshal(newsBytes, &news)
  	if err != nil {
  		return fmt.Errorf("failed to unmarshal news: %w", err)
  	}

  	for _, article := range news.Articles {
  		slog.Info("Article", slog.Time("published_at", article.PublishedAt), slog.String("title", article.Title))
  	}

  	return nil
  }

  type MostFrequentAuthors struct{}

  func (t *MostFrequentAuthors) Execute(context.Context) error {
  	newsBytes, err := os.ReadFile("news.json")
  	if err != nil {
  		return fmt.Errorf("failed to read news: %w", err)
  	}

  	var news News
  	err = json.Unmarshal(newsBytes, &news)
  	if err != nil {
  		return fmt.Errorf("failed to unmarshal news: %w", err)
  	}

  	authors := make(map[string]int)
  	for _, article := range news.Articles {
  		if article.Author == nil {
  			continue
  		}
  		authors[*article.Author]++
  	}

  	for author, count := range authors {
  		slog.Info("Author", slog.String("author", author), slog.Int("count", count))
  	}

  	return nil
  }

  // in main now submit a job, and then visualize it
  /*
  job, err := client.Jobs.Submit(ctx, "process-news",
  	[]workflows.Task{
  		&NewsWorkflow{
  			Category:    "science",
  			MaxArticles: 5,
  		},
  	},
  )
  */
  ```
</CodeGroup>

```plaintext Logs theme={"system"}
News headline published_at=2024-02-15 title="NASA selects ultraviolet astronomy mission but delays its launch two years - SpaceNews"
News headline published_at=2024-02-15 title="SpaceX launches Space Force mission from Cape Canaveral - Orlando Sentinel"
News headline published_at=2024-02-14 title="Saturn's largest moon most likely uninhabitable - Phys.org"
News headline published_at=2024-02-14 title="AI Unveils Mysteries of Unknown Proteins' Functions - Neuroscience News"
News headline published_at=2024-02-14 title="Anthropologists' research unveils early stone plaza in the Andes - Phys.org"
Author article count author="Jeff Foust" count=1
Author article count author="Richard Tribou" count=1
Author article count author="Jeff Renaud" count=1
Author article count author="Neuroscience News" count=1
Author article count author="Science X" count=1
```

<Frame>
  <img alt="Process News Workflow" />

  <img alt="Process News Workflow" />
</Frame>

This workflow consists of four tasks:

| Task                | Dependencies | Description                                                                                                             |
| ------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------- |
| NewsWorkflow        | -            | The root task of the workflow. It spawns the other tasks and sets up the dependencies between them.                     |
| FetchNews           | -            | A task that fetches news articles from the API and writes the results to a file, which is then read by dependent tasks. |
| PrintHeadlines      | FetchNews    | A task that logs the headlines of the news articles.                                                                    |
| MostFrequentAuthors | FetchNews    | A task that counts the number of articles each author has written and logs the result.                                  |

An important aspect is that there is no dependency between the `PrintHeadlines` and `MostFrequentAuthors` tasks. This means they can execute in parallel, which the Tilebox Workflow Orchestrator will do, provided multiple runners are available.

<Tip>
  In this example, the results from `FetchNews` are stored in a file. This is not the recommended method for passing data between tasks. When executing on a distributed cluster, the existence of a file written by a dependent task cannot be guaranteed. Instead, it's better to use a [shared cache](/workflows/run-and-inspect/caches).
</Tip>

## Optional Tasks

By default, if any task in a job fails (after exhausting all [retries](#retry-handling)), the entire job is marked as failed and all remaining queued tasks are canceled. In some workflows though, certain tasks are not critical. Their failure should not prevent the rest of the job from completing. For these cases, you can mark a subtask as **optional**.

An optional task has the following behavior:

* If it **succeeds**, the job continues as normal, there is no difference from a regular task.
* If it **fails**, the job is **not** canceled. Instead:
  * The failed task is marked with the state `FAILED_OPTIONAL` instead of `FAILED`.
  * Tasks that [depend on](#dependencies) the optional task **still execute**, even though the optional task failed.
  * The parent task and the rest of the job continue as normal.

Some scenarios where optional tasks are useful are:

* **Data enrichment**: A task responsible for fetching auxiliary data that is not critical for the job to complete.
* **Reporting**: If a task is a notification or logging task, its failure should not prevent the rest of the job from completing.
* **Fault tolerance**: If a task is known to be flaky and may fail intermittently, marking it as optional can help ensure the job continues to make progress.
* **Aggregation workflows**: If a workflow is composed of multiple independent subtasks, and an aggregation task summarizing the results, not every subtask needs to succeed for the aggregation task to run.
* **Cleanup tasks**: If certain tasks need to always run at the end of a job, for example to send a notification, or to clean up temporary resources, marking the job tasks as optional ensures they always run.

<Note>
  Optional tasks can be combined with [retry handling](#retry-handling). An optional task is only marked as `FAILED_OPTIONAL` after all retries have been exhausted. For example, `context.submit_subtask(FlakyTask(), optional=True, max_retries=3)` will retry up to 3 times before being treated as a failed optional task.
</Note>

### Submitting Optional Tasks

To mark a subtask as optional, use the `optional` parameter when submitting it:

<CodeGroup>
  ```python Python theme={"system"}
  class RootTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          required_step = context.submit_subtask(
              RequiredTask(),
          )
          optional_step = context.submit_subtask(
              FlakyTask(), optional=True
          )
          context.submit_subtask(
              FinalTask(), depends_on=[required_step, optional_step]
          )

  class RequiredTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          # this task may fail, but the job will continue regardless
          context.logger.info("Required task completed")

  class FlakyTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          # this task may fail, but the job will continue regardless
          context.logger.info("Attempting flaky operation")

  class FinalTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          # this task runs even if FlakyTask failed
          context.logger.info("Running final step")
  ```

  ```go Go theme={"system"}
  type RootTask struct{}

  func (t *RootTask) Execute(ctx context.Context) error {
  	requiredStep, err := workflows.SubmitSubtask(ctx, &RequiredTask{})
  	if err != nil {
  		return err
  	}

  	optionalStep, err := workflows.SubmitSubtask(ctx, &FlakyTask{},
  		subtask.WithOptional(),
  	)
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(ctx, &FinalTask{},
  		subtask.WithDependencies(requiredStep, optionalStep),
  	)
  	return err
  }

  type RequiredTask struct{}

  func (t *RequiredTask) Execute(context.Context) error {
  	slog.Info("This task is required to succeed, otherwise the job would stop")
  	return nil
  }

  type FlakyTask struct{}

  func (t *FlakyTask) Execute(context.Context) error {
  	// this task may fail, but the job will continue regardless
  	slog.Info("Attempting flaky operation...")
  	return nil
  }

  type FinalTask struct{}

  func (t *FinalTask) Execute(context.Context) error {
  	// this task runs even if FlakyTask failed
  	slog.Info("Running final step")
  	return nil
  }
  ```
</CodeGroup>

In this example, `FlakyTask` is submitted as an optional subtask. If it fails, `FinalTask` still executes because it depends on an optional task. The job completes successfully. A visualization of such a job is shown below.

<Frame>
  <img alt="Optional Subtasks Workflow" />

  <img alt="Optional Subtasks Workflow" />
</Frame>

### Nested Optional Tasks

When an optional task itself submits subtasks, those subtasks, and also their subtasks recursively, are also considered optional. If any of those tasks fail, all remaining queued tasks that are nested within the same optional root task are automatically **skipped**. This ensures that the failure does not propagate beyond the optional boundary and the parent job continues normally.

<CodeGroup>
  ```python Python theme={"system"}
  class Pipeline(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.submit_subtask(
              OptionalProcessing(), optional=True
          )
          context.submit_subtask(AlwaysRuns())

  class OptionalProcessing(Task):
      def execute(self, context: ExecutionContext) -> None:
          first = context.submit_subtask(Step1())
          context.submit_subtask(Step2(), depends_on=[first])

  class Step1(Task):
      def execute(self, context: ExecutionContext) -> None:
          raise ValueError("something went wrong")

  class Step1A(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Step1A executed successfully")

  class Step1B(Task):
      def execute(self, context: ExecutionContext) -> None:
          raise ValueError("something went wrong")

  class Step1C(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("This will be skipped because Step1B failed")

  class Step2(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("This will be skipped because Step1B failed")

  class AlwaysRuns(Task):
      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("This runs regardless")
  ```

  ```go Go theme={"system"}
  type Pipeline struct{}

  func (t *Pipeline) Execute(ctx context.Context) error {
  	_, err := workflows.SubmitSubtask(ctx, &OptionalProcessing{},
  		subtask.WithOptional(),
  	)
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(ctx, &AlwaysRuns{})
  	return err
  }

  type OptionalProcessing struct{}

  func (t *OptionalProcessing) Execute(ctx context.Context) error {
  	first, err := workflows.SubmitSubtask(ctx, &Step1{})
  	if err != nil {
  		return err
  	}

  	_, err = workflows.SubmitSubtask(ctx, &Step2{},
  		subtask.WithDependencies(first),
  	)
  	return err
  }

  type Step1 struct{}

  func (t *Step1) Execute(context.Context) error {
  	workflows.SubmitSubtask(ctx, &Step1A{})
  	workflows.SubmitSubtask(ctx, &Step1B{})
  	workflows.SubmitSubtask(ctx, &Step1C{})
  	return nil
  }

  type Step1A struct{}

  func (t *Step1A) Execute(context.Context) error {
    slog.Info("Step1A executed successfully")
  	return nil
  }

  type Step1B struct{}

  func (t *Step1B) Execute(context.Context) error {
  	return errors.New("something went wrong")
  }

  type Step1C struct{}

  func (t *Step1C) Execute(context.Context) error {
    slog.Info("This will be skipped because Step1B failed")
  	return nil
  }

  type Step2 struct{}

  func (t *Step2) Execute(context.Context) error {
  	slog.Info("This will be skipped because Step1B failed")
  	return nil
  }

  type AlwaysRuns struct{}

  func (t *AlwaysRuns) Execute(context.Context) error {
  	slog.Info("This runs regardless")
  	return nil
  }
  ```
</CodeGroup>

In this example, `Step1B` fails. Since it's an indirect subtask of the optional `Processing` subtask, both `Step1C` and `Step2` are skipped and `AlwaysRuns` still executes. The job completes successfully.

<Frame>
  <img alt="Optional Subtree Workflow" />

  <img alt="Optional Subtree Workflow" />
</Frame>

If instead `Step1B` was also marked as optional, `Step1C` and `Step2` would still be executed, and only after that `AlwaysRuns` would execute. This means that optional subtasks can have other optional subtasks nested within them.

<Frame>
  <img alt="Optional Subtree Workflow" />

  <img alt="Optional Subtree Workflow" />
</Frame>

## Task Identifiers

A task identifier is a unique string used by the Tilebox Workflow Orchestrator to identify the task. It's used by [runners](/workflows/concepts/runners) to map submitted tasks to a task class and execute them. It also serves as the default name in execution visualizations.

If unspecified, the identifier of a task defaults to the class name. For instance, the identifier of the `PrintHeadlines` task in the previous example is `"PrintHeadlines"`. This is good for prototyping, but not recommended for production, as changing the class name also changes the identifier, which can lead to issues during refactoring. It also prevents different tasks from sharing the same class name.

To address this, Tilebox Workflows offers a way to explicitly specify the identifier of a task. This is done by overriding the `identifier` method of the `Task` class. This method should return a unique string identifying the task. This decouples the task's identifier from its class name, allowing you to change the identifier without renaming the class. It also allows tasks with the same class name to have different identifiers. The `identifier` method can also specify a version number for the task—see the section on [semantic versioning](#semantic-versioning) below for more details.

<CodeGroup title="Overriding the Task Identifier">
  ```python Python theme={"system"}
  class MyTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          pass

  # MyTask has the identifier "MyTask" and the default version of "v0.0"

  class MyTask2(Task):
      @staticmethod
      def identifier() -> tuple[str, str]:
          return "tilebox.com/example_workflow/MyTask", "v1.0"

      def execute(self, context: ExecutionContext) -> None:
          pass

  # MyTask2 has the identifier "tilebox.com/example_workflow/MyTask" and the version "v1.0"
  ```

  ```go Go theme={"system"}
  type MyTask struct{}

  func (t *MyTask) Execute(context.Context) error {
  	return nil
  }

  // MyTask has the identifier "MyTask" and the default version of "v0.0"

  type MyTask2 struct{}

  func (t *MyTask2) Identifier() workflows.TaskIdentifier {
  	return workflows.NewTaskIdentifier("tilebox.com/example_workflow/MyTask", "v1.0")
  }

  func (t *MyTask2) Execute(context.Context) error {
  	return nil
  }

  // MyTask2 has the identifier "tilebox.com/example_workflow/MyTask" and the version "v1.0"
  ```
</CodeGroup>

<Note>
  In python, the `identifier` method must be defined as either a `classmethod` or a `staticmethod`, meaning it can be called without instantiating the class.
</Note>

## Semantic Versioning

As seen in the previous section, the `identifier` method can return a tuple of two strings, where the first string is the identifier and the second string is the version number. This allows for semantic versioning of tasks.

Versioning is important for managing changes to a task's execution method. It allows for new features, bug fixes, and changes while ensuring existing workflows operate as expected. Additionally, it enables multiple versions of a task to coexist, enabling gradual rollout of changes without interrupting production deployments.

You assign a version number by overriding the `identifier` method of the task class. It must return a tuple of two strings: the first is the [identifier](#task-identifiers) and the second is the version number, which must match the pattern `vX.Y` (where `X` and `Y` are non-negative integers). `X` is the major version number and `Y` is the minor version.

For example, this task has the identifier `"tilebox.com/example_workflow/MyTask"` and the version `"v1.3"`:

<CodeGroup title="Overriding the Task Identifier">
  ```python Python theme={"system"}
  class MyTask(Task):
      @staticmethod
      def identifier() -> tuple[str, str]:
          return "tilebox.com/example_workflow/MyTask", "v1.3"

      def execute(self, context: ExecutionContext) -> None:
          pass
  ```

  ```go Go theme={"system"}
  type MyTask struct{}

  func (t *MyTask) Identifier() workflows.TaskIdentifier {
  	return workflows.NewTaskIdentifier("tilebox.com/example_workflow/MyTask", "v1.3")
  }

  func (t *MyTask) Execute(context.Context) error {
  	return nil
  }
  ```
</CodeGroup>

When a task is submitted as part of a job, the version from which it's submitted is recorded and may differ from the version on the runner executing the task.

When runners execute a task, they require a registered task with a matching identifier and compatible version number. A compatible version is where the major version number on the runner matches that of the submitted task, and the minor version number on the runner is equal to or greater than that of the submitted task.

Examples of compatible version numbers include:

* `MyTask` is submitted as part of a job. The version is `"v1.3"`.
* A runner with version `"v1.3"` of `MyTask` would execute this task.
* A runner with version `"v1.5"` of `MyTask` would also execute this task.
* A runner with version `"v1.2"` of `MyTask` would not execute this task, as its minor version is lower than that of the submitted task.
* A runner with version `"v2.5"` of `MyTask` would not execute this task, as its major version differs from that of the submitted task.

## Conclusion

Tasks form the foundation of Tilebox Workflows. By understanding how to create and manage tasks, you can leverage Tilebox's capabilities to automate and optimize your workflows. Experiment with defining your own tasks, utilizing subtasks, managing dependencies, and employing semantic versioning to develop robust and efficient workflows.


# Workflow Releases
Source: https://docs.tilebox.com/workflows/concepts/workflow-releases

Understand workflow releases, release artifacts, and how release runners execute deployed Python workflow projects.

A workflow is a set of interrelated tasks. You can run those tasks directly without registering the workflow with Tilebox. Registering a workflow with the Tilebox API gives it a stable slug, which lets you publish immutable release artifacts to it and deploy a release to one or more clusters.

For a new Python release project, `tilebox workflow init` creates the workflow object and scaffolds the local project files used to build releases.

That release path enables [release runners](/workflows/concepts/runners#release-runners). Release runners operate on a cluster, pick up all the releases deployed to that cluster, and execute tasks. This provides an easy way of deploying workflows to a compute cluster, including a quick and agent-accessible iteration loop: change code, publish a release, deploy it, run a job, and inspect the result.

## Workflows and releases

A workflow is the long-lived object referred to by slug. A release is one concrete version of that workflow. The release is immutable, so a later code change creates a new release instead of modifying the old one. You can deploy the same release to one or multiple clusters. Release runners on those clusters then pick up that release and run tasks registered by it.

<Frame>
  <img alt="Workflow slug with multiple releases, release artifacts, and cluster deployments" />

  <img alt="Workflow slug with multiple releases, release artifacts, and cluster deployments" />
</Frame>

Use this model when you want reproducible workflow execution. You can inspect which release is deployed to a cluster, promote a known release to another cluster, or retry a failed job after deploying a compatible fix.

## Release artifacts

The release artifact is built from the files selected by `tilebox.workflow.toml`. The build command resolves include patterns, applies exclude patterns and `.gitignore` when enabled, creates a deterministic `.tar.zst` archive, and validates the runtime by discovering registered tasks.

The artifact should contain code and small configuration. Keep downloaded data, model checkpoints, generated caches, and local virtual environments out of the release. If a workflow needs large runtime assets, fetch them lazily from the task code into a runner-local cache.

## Task registrations

Task registrations are discovered from the configured Python runner object or command during release validation. The discovered task identifiers are stored in the release content and later advertised by release runners.

For a reusable Python workflow project, define a `Runner` object:

```python Python theme={"system"}
# my_workflow/runner.py
from tilebox.workflows import ExecutionContext, Runner, Task


class FirstTask(Task):
    def execute(self, context: ExecutionContext) -> None:
        ...


class SecondTask(Task):
    def execute(self, context: ExecutionContext) -> None:
        ...


runner = Runner(tasks=[FirstTask, SecondTask])
```

Then point `tilebox.workflow.toml` at that object:

```toml theme={"system"}
[workflow]
slug = "my-workflow"
root = "."
runner = "my_workflow.runner:runner"
```

## Cluster deployments

A cluster deployment maps a workflow release to a cluster. A release runner can run multiple deployed releases for the same cluster and updates its task registrations when cluster deployments change.

Deploying, updating, or removing a release deployment changes what the release runner can execute. It does not require rebuilding the runner process itself.

## Fixing failed jobs

If a job fails because of a bug in task code, publish a compatible fixed release and deploy it to the same cluster before retrying the job. Keep the task identifier name, major version, and input schema compatible when you want the existing failed job to resume from failed tasks.

```bash theme={"system"}
tilebox workflow publish-release --json
tilebox workflow deploy-release --latest --cluster dev-cluster --json
tilebox job retry <job-id> --json
```


# Tilebox Workflows
Source: https://docs.tilebox.com/workflows/introduction

Run space data workflows across local, on-premises, and cloud environments with agent-friendly iteration loops.

<Card title="Run your first workflow" icon="play" href="/guides/workflows/run-your-first-workflow">
  Want to learn by example? Run your first workflow by creating a task, starting a runner, submitting a job, and inspecting the result.
</Card>

Tilebox Workflows is a parallel processing engine for space data operations. It helps you turn processing steps such as fetching scenes, validating products, generating previews, running models, and publishing outputs into [tasks](/workflows/concepts/tasks) that can run across local machines, on-premises systems, and cloud environments.

<Frame>
  <img alt="Tilebox Workflows architecture showing workflow code submitted as a job, Tilebox tracking queued tasks and job state, and a runner executing matching tasks" />

  <img alt="Tilebox Workflows architecture showing workflow code submitted as a job, Tilebox tracking queued tasks and job state, and a runner executing matching tasks" />
</Frame>

You submit a [job](/workflows/concepts/jobs) from workflow code, and Tilebox tracks the resulting task graph while [runners](/workflows/concepts/runners) execute eligible work on the selected [cluster](/workflows/concepts/clusters). During development, you can run tasks directly from your own script or service. For reproducible deployment, you can publish [workflow releases](/workflows/concepts/workflow-releases), deploy them to clusters, and let release runners execute the deployed code.

This model also gives AI coding agents a practical iteration loop: edit workflow code, publish a release, deploy it to a development cluster, submit a test job, and inspect logs, traces, and job state before making the next change.

## Get Started with Tilebox Workflows

<Columns>
  <Card title="Create tasks" icon="laptop-code" href="/workflows/concepts/tasks">
    Define task classes, inputs, subtasks, dependencies, retries, and identifiers.
  </Card>

  <Card title="Submit jobs" icon="diagram-project" href="/workflows/concepts/jobs">
    Submit a root task to a cluster and let runners execute the resulting task graph.
  </Card>

  <Card title="Set up a runner" icon="list-check" href="/workflows/concepts/runners">
    Learn how to set up a runner in order to execute tasks.
  </Card>

  <Card title="Build workflow releases" icon="box-open" href="/workflows/concepts/workflow-releases">
    Package workflow projects into immutable releases.
  </Card>

  <Card title="Deploy releases to clusters" icon="rocket" href="/workflows/build-and-deploy/cluster-deployments">
    Map releases to clusters and run them with `tilebox runner start`.
  </Card>

  <Card title="Inspect workflow runs" icon="eye" href="/workflows/run-and-inspect/introduction">
    Use logs, traces, and job state to debug workflows across runners.
  </Card>
</Columns>

## Terminology

Before exploring Tilebox Workflows in depth, familiarize yourself with some common terms used throughout this section.

<AccordionGroup>
  <Accordion title="Tasks">
    A Task is the smallest unit of work, designed to perform a specific operation. Each task represents a distinct operation or process that can be executed, such as processing data, performing calculations, or managing resources. Tasks can operate independently or as components of a more complex set of connected tasks known as a Workflow. Tasks are defined by their code, inputs, and dependencies on other tasks. To create tasks, you need to define the input parameters and specify the action to be performed during execution.
  </Accordion>

  <Accordion title="Jobs">
    A job is a specific execution of a workflow with designated input parameters. It consists of one or more tasks that can run in parallel or sequentially, based on their dependencies. Submitting a job involves creating a root task with specific input parameters, which may trigger the execution of other tasks within the same job.
  </Accordion>

  <Accordion title="Runners">
    Runners are processes that execute workflow tasks for a cluster. Direct runners register task classes from a standalone script, service, or binary that connects to the Tilebox API through the SDK. Release runners are started by the Tilebox CLI and load task registrations from Python workflow releases deployed to their cluster.
  </Accordion>

  <Accordion title="Workflow releases">
    A workflow release is an immutable package of a workflow project. It includes source files, the command or runner object used to start the workflow runtime, and the task identifiers discovered during release validation.
  </Accordion>

  <Accordion title="Clusters">
    Clusters group runners and receive workflow release deployments. Jobs are submitted to a cluster, and only runners assigned to that cluster can claim the job's tasks. A release runner can run multiple releases that are deployed to its cluster.
  </Accordion>

  <Accordion title="Caches">
    Caches are shared storage that enable data storage and retrieval across tasks within a single job. They store intermediate results and share data among tasks, enabling distributed computing and reducing redundant data processing.
  </Accordion>

  <Accordion title="Observability">
    Observability refers to the feature set in Tilebox Workflows that provides visibility into the execution of tasks and jobs. Tools like tracing and logging allow users to monitor performance, diagnose issues, and gain insights into job operations, enabling efficient troubleshooting and optimization.
  </Accordion>
</AccordionGroup>


# Caches
Source: https://docs.tilebox.com/workflows/run-and-inspect/caches

Sharing data between tasks is crucial for workflows, especially in satellite imagery processing, where large datasets are the norm. Tilebox Workflows offers a straightforward API for storing and retrieving data from a shared cache.

<Warning>
  The cache API is currently experimental and may undergo changes in the future. Many more features and new [backends](#cache-backends) are on the roadmap. There might be breaking changes to the Cache API in the future.
</Warning>

Caches are configured at the [runner](/workflows/concepts/runners) level. Because runners can be deployed across multiple locations, caches must be accessible from all runners contributing to a workflow.

Currently, the default cache implementation uses a Google Cloud Storage bucket, providing a scalable method to share data between tasks. For quick prototyping and local development, you can also use a local file system cache, which is included by default.

If needed, you can create your own cache backend by implementing the `Cache` interface.

## Configuring a Cache

You can configure a cache while creating a runner by passing a cache instance to the `cache` parameter. To use an in-memory cache, use `tilebox.workflows.cache.InMemoryCache`. This implementation is helpful for local development and quick testing. For alternatives, see the supported [cache backends](#cache-backends).

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import InMemoryCache

  client = Client()
  runner = client.runner(
      tasks=[...],
      cache=InMemoryCache(),
  )
  ```
</CodeGroup>

By configuring such a cache, the `context` object that is passed to the execution of each task gains access to a `job_cache`
attribute that can be used to [store and retrieve data](#storing-and-retrieving-data) from the cache.

## Cache Backends

Tilebox Workflows comes with four cache implementations out of the box, each backed by a different storage system.

<Columns>
  <Card title="Google Cloud Storage" icon="google" href="#google-storage-cache" />

  <Card title="Amazon S3" icon="aws" href="#amazon-s3-cache" />

  <Card title="Local File System" icon="folder-open" href="#local-file-system-cache" />

  <Card title="In-Memory" icon="memory" href="#in-memory-cache" />
</Columns>

### Google Storage Cache

A cache implementation backed by a Google cloud Storage bucket to store and retrieve data. It's a reliable method for sharing data across tasks, suitable for production environments. You will need access to a GCP project and a bucket.

The Tilebox Workflow orchestrator uses the official Python Client for Google Cloud Storage. To set up authentication, refer to the <Link href="https://cloud.google.com/storage/docs/reference/libraries#authentication">official documentation</Link>.

<CodeGroup title="GoogleStorageCache">
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import GoogleStorageCache
  from google.cloud.storage import Client as StorageClient

  storage_client = StorageClient(project="gcp-project")
  bucket = storage_client.bucket("cache-bucket")

  client = Client()
  runner = client.runner(
      tasks=[...],
      cache=GoogleStorageCache(bucket, prefix="jobs"),
  )
  ```
</CodeGroup>

<Tip>
  The `prefix` parameter is optional and can be used to set a common prefix for all cache keys, which helps organize objects within a bucket when re-using the same bucket for other purposes.
</Tip>

### Amazon S3 Cache

A cache implementation backed by an Amazon S3 bucket to store and retrieve data. Like the Google Cloud Storage option, it's reliable and scalable for production use.

Tilebox utilizes the `boto3` library to communicate with Amazon S3. For the necessary authentication setup, refer to [its documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration).

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import AmazonS3Cache

  client = Client()
  runner = client.runner(
      tasks=[...],
      cache=AmazonS3Cache("my-bucket-name", prefix="jobs")
  )
  ```
</CodeGroup>

<Tip>
  The `prefix` parameter is optional and can be used to set a common prefix for all cache keys, which helps organize objects within a bucket when re-using the same bucket for other purposes.
</Tip>

### Local File System Cache

A cache implementation backed by a local file system. It's suitable for quick prototyping and local development, assuming all runners share the same machine or access the same file system.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import LocalFileSystemCache

  client = Client()
  runner = client.runner(
      tasks=[...],
      cache=LocalFileSystemCache("/path/to/cache/directory"),
  )
  ```
</CodeGroup>

<Tip>
  Local file system caches can also be used in conjunction with network attached storage or similar tools, making it a viable option also for distributed setups.
</Tip>

### In-Memory Cache

A simple in-memory cache useful for quick prototyping and development. The data is not shared between runners and is lost upon runner restarts. Use this cache only for workflows executed on a single runner.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.cache import InMemoryCache

  client = Client()
  runner = client.runner(
      tasks=[...],
      cache=InMemoryCache(),
  )
  ```
</CodeGroup>

## Data Isolation

Caches are isolated per job, meaning that each job's cache data is only accessible to tasks within that job. This setup prevents key conflicts between different job executions. Currently, accessing cache data of other jobs is not supported.

<Note>
  The capability to share data across jobs is planned for future updates. This feature will be beneficial for real-time processing workflows or workflows requiring auxiliary data from external sources.
</Note>

## Storing and Retrieving Data

The job cache can be accessed via the `ExecutionContext` passed to a tasks `execute` function. This [`job_cache`](/api-reference/python/tilebox.workflows/ExecutionContext.job_cache) object provides methods to handle data storage and retrieval from the cache. The specifics of data storage depend on the chosen cache backend.

The cache API is designed to be simple and can handle all types of data, supporting binary data in the form of `bytes`, identified by `str` cache keys. This allows for storing many different data types, such as pickled Python objects, serialized JSON, UTF-8, or binary data.

The following snippet illustrates storing and retrieving data from the cache.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class ProducerTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          # store data in the cache
          context.job_cache["data"] = b"my_binary_data_to_store"
          context.submit_subtask(ConsumerTask())

  class ConsumerTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          data = context.job_cache["data"]
          context.logger.info("Read data from cache", data=data.decode())
  ```
</CodeGroup>

In this example, data stored under the key `"data"` can be any size that fits the cache backend constraints. Ensure the key remains unique within the job's scope to avoid conflicts.

To test the workflow, you can start a local runner using the `InMemoryCache` backend. Then, submit a job to execute the `ProducerTask` and inspect the logs emitted by the `ConsumerTask`.

<CodeGroup>
  ```python Python theme={"system"}
  # submit a job to test our workflow
  job_client = client.jobs()
  job_client.submit("testing-cache-access", ProducerTask())

  # start a runner to execute it
  runner = client.runner(
      tasks=[ProducerTask, ConsumerTask],
      cache=LocalFileSystemCache("/path/to/cache/directory"),
  )
  runner.run_forever()
  ```
</CodeGroup>

```plaintext Logs theme={"system"}
Read data from cache data=my_binary_data_to_store
```

## Groups and Hierarchical Keys

The cache API supports groups and hierarchical keys, analogous to directories and files in a file system. Groups help organize cache keys hierarchically, preventing key conflicts and allowing data to be structured better. Additionally, groups are iterable, enabling retrieval of all keys within the group. This feature is useful when multiple tasks create cache data, and a later task needs to list all produced data by earlier tasks.

The following code shows an example of how cache groups can be used.

<CodeGroup>
  ```python Python {39,44-45} theme={"system"}
  from tilebox.workflows import Task, ExecutionContext
  import random

  class CacheGroupDemo(Task):
      n: int

      def execute(self, context: ExecutionContext):
          # define a cache group key for subtasks
          group_key = "random_numbers"
          produce_numbers = context.submit_subtask(
              ProduceRandomNumbers(self.n, group_key)
          )
          sum_task = context.submit_subtask(
              PrintSum(group_key),
              depends_on=[produce_numbers],
          )

  class ProduceRandomNumbers(Task):
      n: int
      group_key: str

      def execute(self, context: ExecutionContext):
          for i in range(self.n):
              context.submit_subtask(ProduceRandomNumber(i, self.group_key))

  class ProduceRandomNumber(Task):
      index: int
      group_key: str

      def execute(self, context: ExecutionContext) -> None:
          number = random.randint(0, 100)
          group = context.job_cache.group(self.group_key)
          group[f"key_{self.index}"] = str(number).encode()

  class PrintSum(Task):
      group_key: str

      def execute(self, context: ExecutionContext) -> None:
          group = context.job_cache.group(self.group_key)

          # PrintSum doesn't know how many numbers were produced,
          # instead it iterates over all keys in the cache group
          numbers = []
          for key in group:  # iterate over all stored numbers
              number = group[key]  # read data from cache
              numbers.append(int(number.decode()))  # convert bytes back to int

          context.logger.info("Computed sum of all numbers", total=sum(numbers))
  ```
</CodeGroup>

Submitting a job of the `CacheGroupDemo` and running it with a runner can be done as follows:

<CodeGroup>
  ```python Python theme={"system"}
  # submit a job to test our workflow
  job_client = client.jobs()
  job_client.submit("cache-groups", CacheGroupDemo(5))

  # start a runner to execute it
  runner = client.runner(
      tasks=[CacheGroupDemo, ProduceRandomNumbers, ProduceRandomNumber, PrintSum],
      cache=LocalFileSystemCache("/path/to/cache/directory"),
  )
  runner.run_forever()
  ```
</CodeGroup>

```plaintext Logs theme={"system"}
Computed sum of all numbers total=284
```


# Axiom
Source: https://docs.tilebox.com/workflows/run-and-inspect/integrations/axiom

Export Tilebox workflow logs and traces to Axiom datasets in addition to Tilebox Console.

Tilebox can export workflow telemetry to [Axiom](https://axiom.co/) through OTLP. Use this when your team already analyzes logs and traces in Axiom or wants long-term telemetry in Axiom datasets.

Built-in Tilebox Console observability does not require Axiom. Axiom export is optional and additive.

<Frame>
  <img alt="Tilebox is pre-integrated with Axiom" />
</Frame>

## Configure Axiom export

Create Axiom datasets for logs and traces and an API key with ingest permissions. Then configure export when the runner process starts.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.observability.logging import configure_otel_logging_axiom
  from tilebox.workflows.observability.tracing import configure_otel_tracing_axiom

  from my_workflow import ProcessScene

  configure_otel_tracing_axiom(
      service="sentinel-2-runner",
      dataset="workflow-traces",
      api_key="<axiom-api-key>",
  )
  configure_otel_logging_axiom(
      service="sentinel-2-runner",
      dataset="workflow-logs",
      api_key="<axiom-api-key>",
  )

  client = Client(name="sentinel-2-runner")
  runner = client.runner(tasks=[ProcessScene])
  runner.run_forever()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log/slog"

  	"github.com/tilebox/tilebox-go/observability"
  	"github.com/tilebox/tilebox-go/observability/logger"
  	"github.com/tilebox/tilebox-go/observability/tracer"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"go.opentelemetry.io/otel"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "processing scene")
  	return nil
  }

  func main() {
  	ctx := context.Background()
  	service := &observability.Service{Name: "sentinel-2-runner"}
  	apiKey := "<axiom-api-key>"

  	traceProvider, shutdownTracer, err := tracer.NewAxiomProvider(ctx, service, "workflow-traces", apiKey)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to configure Axiom tracing", slog.Any("error", err))
  		return
  	}
  	defer shutdownTracer(ctx)
  	otel.SetTracerProvider(traceProvider)

  	logHandler, shutdownLogger, err := logger.NewAxiomHandler(ctx, service, "workflow-logs", apiKey,
  		logger.WithLevel(slog.LevelInfo),
  	)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to configure Axiom logging", slog.Any("error", err))
  		return
  	}
  	defer shutdownLogger(ctx)
  	slog.SetDefault(logger.New(logHandler))

  	client := workflows.NewClient()
  	runner, err := client.NewTaskRunner(ctx)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to create runner", slog.Any("error", err))
  		return
  	}

  	if err := runner.RegisterTasks(&ProcessScene{}); err != nil {
  		slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err))
  		return
  	}

  	runner.Run(ctx)
  }
  ```
</CodeGroup>

## Environment variables

You can omit credentials from code by setting environment variables:

| Variable               | Used by                          |
| ---------------------- | -------------------------------- |
| `AXIOM_API_KEY`        | log and trace export             |
| `AXIOM_LOGS_DATASET`   | `configure_otel_logging_axiom()` |
| `AXIOM_TRACES_DATASET` | `configure_otel_tracing_axiom()` |

<CodeGroup>
  ```python Python theme={"system"}
  configure_otel_tracing_axiom(service="sentinel-2-runner")
  configure_otel_logging_axiom(service="sentinel-2-runner")
  ```

  ```go Go theme={"system"}
  traceProvider, shutdownTracer, err := tracer.NewAxiomProviderFromEnv(ctx, service)
  if err != nil {
  	slog.ErrorContext(ctx, "failed to configure Axiom tracing", slog.Any("error", err))
  	return
  }
  defer shutdownTracer(ctx)
  otel.SetTracerProvider(traceProvider)

  logHandler, shutdownLogger, err := logger.NewAxiomHandlerFromEnv(ctx, service,
  	logger.WithLevel(slog.LevelInfo),
  )
  if err != nil {
  	slog.ErrorContext(ctx, "failed to configure Axiom logging", slog.Any("error", err))
  	return
  }
  defer shutdownLogger(ctx)
  slog.SetDefault(logger.New(logHandler))
  ```
</CodeGroup>

## Existing Axiom screenshots

<Frame>
  <img alt="Tilebox workflow logs in Axiom" />

  <img alt="Tilebox workflow logs in Axiom" />
</Frame>

<Frame>
  <img alt="Tilebox workflow traces in Axiom" />

  <img alt="Tilebox workflow traces in Axiom" />
</Frame>


# OpenTelemetry
Source: https://docs.tilebox.com/workflows/run-and-inspect/integrations/open-telemetry

Export Tilebox workflow logs and traces to any OTLP-compatible backend in addition to Tilebox Console.

Tilebox uses OpenTelemetry data models for workflow telemetry. Built-in Tilebox observability works without any OpenTelemetry setup, but you can add OTLP export when you need the same logs and traces in another backend.

## Configure OTLP export

Call the configuration functions when the runner process starts, before creating the client or runner.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client
  from tilebox.workflows.observability.logging import configure_otel_logging
  from tilebox.workflows.observability.tracing import configure_otel_tracing

  from my_workflow import ProcessScene

  configure_otel_tracing(
      service="sentinel-2-runner",
      endpoint="http://localhost:4318/v1/traces",
      headers={"Authorization": "Bearer <token>"},
  )
  configure_otel_logging(
      service="sentinel-2-runner",
      endpoint="http://localhost:4318/v1/logs",
      headers={"Authorization": "Bearer <token>"},
  )

  client = Client(name="sentinel-2-runner")
  runner = client.runner(tasks=[ProcessScene])
  runner.run_forever()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log/slog"

  	"github.com/tilebox/tilebox-go/observability"
  	"github.com/tilebox/tilebox-go/observability/logger"
  	"github.com/tilebox/tilebox-go/observability/tracer"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"go.opentelemetry.io/otel"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "processing scene")
  	return nil
  }

  func main() {
  	ctx := context.Background()
  	service := &observability.Service{Name: "sentinel-2-runner"}

  	traceProvider, shutdownTracer, err := tracer.NewOtelProvider(ctx, service,
  		tracer.WithEndpointURL("http://localhost:4318/v1/traces"),
  		tracer.WithHeaders(map[string]string{"Authorization": "Bearer <token>"}),
  	)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to configure tracing", slog.Any("error", err))
  		return
  	}
  	defer shutdownTracer(ctx)
  	otel.SetTracerProvider(traceProvider)

  	logHandler, shutdownLogger, err := logger.NewOtelHandler(ctx, service,
  		logger.WithEndpointURL("http://localhost:4318/v1/logs"),
  		logger.WithHeaders(map[string]string{"Authorization": "Bearer <token>"}),
  		logger.WithLevel(slog.LevelInfo),
  	)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to configure logging", slog.Any("error", err))
  		return
  	}
  	defer shutdownLogger(ctx)
  	slog.SetDefault(logger.New(logHandler))

  	client := workflows.NewClient()
  	runner, err := client.NewTaskRunner(ctx)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to create runner", slog.Any("error", err))
  		return
  	}

  	if err := runner.RegisterTasks(&ProcessScene{}); err != nil {
  		slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err))
  		return
  	}

  	runner.Run(ctx)
  }
  ```
</CodeGroup>

If the endpoint does not include `/v1/traces` or `/v1/logs`, the Python SDK adds the correct path automatically.

## Environment variables

You can omit endpoint and interval arguments by setting environment variables:

| Variable               | Used by                       |
| ---------------------- | ----------------------------- |
| `OTEL_TRACES_ENDPOINT` | `configure_otel_tracing()`    |
| `OTEL_LOGS_ENDPOINT`   | `configure_otel_logging()`    |
| `OTEL_EXPORT_INTERVAL` | tracing and logging exporters |

`OTEL_EXPORT_INTERVAL` accepts durations such as `5s`, `30s`, or `2m`.

## Local collector example

For local trace testing, run Jaeger with OTLP HTTP enabled:

```bash theme={"system"}
docker run --rm --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.9.0
```

Then configure tracing with `endpoint="http://localhost:4318"` and open the Jaeger UI at [http://localhost:16686](http://localhost:16686).


# Inspect workflow runs
Source: https://docs.tilebox.com/workflows/run-and-inspect/introduction

Inspect workflow logs, traces, task status, and runner behavior while jobs execute.

Tilebox Workflows gives each job a live observability record. As runners execute work, Tilebox captures logs, traces, task status, and runner context. You can follow a job from the root task through its subtasks, inspect failures, and compare slow steps across distributed runners.

<Frame>
  <img alt="Job Execution Trace View" />

  <img alt="Job Execution Trace View" />
</Frame>

Use the built-in view for day-to-day debugging and operations. Add structured log fields and custom spans when you need more detail inside your task code.

<Columns>
  <Card title="Logging" icon="rectangle-terminal" href="/workflows/run-and-inspect/logging" />

  <Card title="Tracing" icon="chart-gantt" href="/workflows/run-and-inspect/tracing" />

  <Card title="Query telemetry" icon="magnifying-glass-chart" href="/workflows/run-and-inspect/query-telemetry" />

  <Card title="Integrations" icon="plug" href="/workflows/run-and-inspect/integrations/open-telemetry" />
</Columns>

## How Tilebox observes workflows

A submitted job starts a trace. Each task run creates a span, and custom spans sit under the task that creates them. Log records emitted from task code attach to active spans, which connects messages to timing data.

Tilebox adds job, task, runner, and service metadata to telemetry records. This data helps you filter by job, inspect a single task run, or compare work across runners.

## Observability example

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client, ExecutionContext, Task

  class ProcessScene(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Processing scene", scene_id=self.scene_id)

          with context.tracer.span("plan-subtasks"):
              thumbnail = context.submit_subtask(BuildThumbnail(scene_id=self.scene_id))
              context.submit_subtask(PublishScene(scene_id=self.scene_id), depends_on=[thumbnail])

  class BuildThumbnail(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Building thumbnail", scene_id=self.scene_id)

  class PublishScene(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Publishing scene", scene_id=self.scene_id)

  client = Client(name="sentinel-2-runner")
  runner = client.runner(tasks=[ProcessScene, BuildThumbnail, PublishScene])
  runner.run_forever()
  ```

  ```go Go theme={"system"}
  package tasks

  import (
  	"context"
  	"fmt"
  	"log/slog"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  	"github.com/tilebox/tilebox-go/workflows/v1/subtask"
  )

  type ProcessScene struct {
  	SceneID string
  }

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "processing scene", slog.String("scene_id", t.SceneID))

  	return workflows.WithSpan(ctx, "plan-subtasks", func(ctx context.Context) error {
  		thumbnail, err := workflows.SubmitSubtask(ctx, &BuildThumbnail{SceneID: t.SceneID})
  		if err != nil {
  			return fmt.Errorf("failed to submit thumbnail subtask: %w", err)
  		}

  		_, err = workflows.SubmitSubtask(ctx, &PublishScene{SceneID: t.SceneID}, subtask.WithDependencies(thumbnail))
  		if err != nil {
  			return fmt.Errorf("failed to submit publish subtask: %w", err)
  		}

  		return nil
  	})
  }

  type BuildThumbnail struct {
  	SceneID string
  }

  func (t *BuildThumbnail) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "building thumbnail", slog.String("scene_id", t.SceneID))
  	return nil
  }

  type PublishScene struct {
  	SceneID string
  }

  func (t *PublishScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "publishing scene", slog.String("scene_id", t.SceneID))
  	return nil
  }
  ```
</CodeGroup>

The parent task, spawned subtasks, task logs, and spans share the same job trace. This keeps orchestration and task-level work connected.

## Integrate with external observability platforms

If your team uses another observability platform, configure [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) export in the runner process. Tilebox keeps the workflow state, while your platform receives the same logs and traces for alerting, long-term storage, or analysis.


# Logging
Source: https://docs.tilebox.com/workflows/run-and-inspect/logging

Emit structured task logs and tune how workflow clients export log records.

Tilebox collects workflow logs automatically. When a runner is created from a `Client`, logs emitted through the task execution context are exported to Tilebox and correlated with the active job, task, and trace.

<Frame>
  <img alt="Job Logs View" />

  <img alt="Job Logs View" />
</Frame>

## Write task logs

Use `context.logger` inside `Task.execute`. It supports standard log levels and structured attributes.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext, Task

  class ProcessScene(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          context.logger.info("Started scene processing", scene_id=self.scene_id)

          log = context.logger.bind(component="atmospheric-correction")
          log.debug("Fetching auxiliary data")

          try:
              # process the scene
              log.info("Scene processed", output_format="cog")
          except Exception:
              context.logger.exception("Scene processing failed", scene_id=self.scene_id)
              raise
  ```

  ```go Go theme={"system"}
  package tasks

  import (
  	"context"
  	"log/slog"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "started scene processing", slog.String("scene_id", "S2A_001"))
  	slog.DebugContext(ctx, "fetching auxiliary data", slog.String("component", "atmospheric-correction"))
  	return nil
  }
  ```
</CodeGroup>

Structured attributes become searchable log attributes. Bind shared attributes to a logger when records need the same context.

## What Tilebox adds automatically

Logs emitted from a task include workflow metadata without extra code. Tilebox attaches job ID, job name, task ID, task display name, parent task data, task identifier name and version, runner service data, SDK version, host, OS, and process ID. When a log is emitted inside an active span, Tilebox also attaches `trace_id` and `span_id`.

Logs are also added as events on the active trace span, so a trace view can show important log messages inline with task execution.

## Configure local console logging

Built-in Tilebox export does not require configuration. For local development, add a console handler to print Tilebox workflow logs to standard output.

<CodeGroup>
  ```python Python theme={"system"}
  import logging

  from tilebox.workflows import Client
  from tilebox.workflows.observability.logging import configure_console_logging

  from my_workflow import ProcessScene

  configure_console_logging(level=logging.DEBUG)

  client = Client()
  runner = client.runner(tasks=[ProcessScene])
  runner.run_forever()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"log/slog"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	slog.InfoContext(ctx, "processing scene")
  	return nil
  }

  func main() {
  	ctx := context.Background()
  	workflows.ConfigureConsoleLogging(slog.LevelDebug)

  	client := workflows.NewClient()
  	runner, err := client.NewTaskRunner(ctx)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to create runner", slog.Any("error", err))
  		return
  	}

  	if err := runner.RegisterTasks(&ProcessScene{}); err != nil {
  		slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err))
  		return
  	}

  	runner.Run(ctx)
  }
  ```
</CodeGroup>

`configure_console_logging()` and `workflows.ConfigureConsoleLogging()` are process-wide for Tilebox workflow loggers. Use them for local runs and debugging distributed runners.

## Configure the client log level

Use `Client.configure_logging()` to choose which task and runner logs a client exports to Tilebox.

<CodeGroup>
  ```python Python theme={"system"}
  import logging

  from tilebox.workflows import Client

  client = Client(name="sentinel-2-runner")

  # Export task logs at DEBUG and internal runner logs at INFO.
  client.configure_logging(level=logging.DEBUG, runner_level=logging.INFO)
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"log/slog"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
  	workflows.ConfigureConsoleLogging(slog.LevelDebug)
  	client := workflows.NewClient()
  	_ = client
  }
  ```
</CodeGroup>

The Python `level` argument applies to logs emitted with `context.logger`. The optional `runner_level` argument applies to internal runner logs. If `runner_level` is omitted, it uses the same value as `level`. In Go, `workflows.ConfigureConsoleLogging()` sets the local console log level, and `workflows.NewClient()` configures Tilebox workflow log export.

## Query logs

You can retrieve logs for a job through the jobs client. Python results can also be converted to a pandas DataFrame.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  job = client.jobs().submit("process-scene", ProcessScene(scene_id="S2A_001"))

  logs = client.jobs().query_logs(job)
  for record in logs:
      print(record.time, record.severity_text, record.body, record.attributes)

  df = logs.to_pandas()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"fmt"
  	"log/slog"
  	"time"

  	"github.com/google/uuid"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := workflows.NewClient()
  	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  	logs, err := workflows.Collect(
  		client.Jobs.QueryLogs(ctx, jobID, workflows.WithSortDirection(workflows.Ascending)),
  	)
  	if err != nil {
  		slog.ErrorContext(ctx, "failed to query job logs", slog.Any("error", err))
  		return
  	}

  	for _, record := range logs {
  		fmt.Printf("%s %-5s %s\n",
  			record.Time.Format(time.RFC3339Nano),
  			record.Level,
  			record.Body,
  		)
  	}
  }
  ```
</CodeGroup>

See [Query telemetry](/workflows/run-and-inspect/query-telemetry) for the log and span query APIs.

## Export to another backend

Tilebox stores logs by default. To export logs to your own observability backend as well, configure an [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) integration when the runner process starts.


# Progress
Source: https://docs.tilebox.com/workflows/run-and-inspect/progress

Report and visualize user-defined progress indicators during job execution to provide visibility into completion status and the estimated remaining time.

Tilebox supports user-defined progress indicators during the execution of a job. This can be useful to provide visibility into the execution and the expected duration of a job, especially for longer running jobs.

<Frame>
  <img alt="Tilebox Workflows progress indicators" />

  <img alt="Tilebox Workflows progress indicators" />
</Frame>

## Tracking Progress

Progress indicators in Tilebox use a `done` / `total` model. Tasks can increase a `total` value to specify the total work to be done, and the same or any other task can increase a `done` counter to track the amount of work that has already been completed.

Progress tracking is always done at a task level. Each task can report its progress updates, as increases in `done` and `total` independently, and the job's total progress is the sum of all tasks' progress.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Task, ExecutionContext

  class ProgressRootTask(Task):
      n: int

      def execute(self, context: ExecutionContext) -> None:
          # report that 10 units of work need to be done
          context.progress().add(self.n)  # [!code ++]

          for _ in range(self.n):
              context.submit_subtask(ProgressSubTask())

  class ProgressSubTask(Task):
      def execute(self, context: ExecutionContext) -> None:
          # report that one unit of work has been completed
          context.progress().done(1)  # [!code ++]
  ```

  ```go Go theme={"system"}
  type ProgressRootTask struct {
  	N int
  }

  func (t *ProgressRootTask) Execute(ctx context.Context) error {
      // report that 10 units of work need to be done
  	err := workflows.DefaultProgress().Add(ctx, uint64(t.N)) // [!code ++]
  	if err != nil {
  		return err
  	}

  	for range t.N {
  		_, err := workflows.SubmitSubtask(ctx, &ProgressSubTask{})
  		if err != nil {
  			return err
  		}
  	}

  	return nil
  }

  type ProgressSubTask struct{}

  func (t *ProgressSubTask) Execute(ctx context.Context) error {
      // report that one unit of work has been completed
  	err := workflows.DefaultProgress().Done(ctx, 1) // [!code ++]
  	if err != nil {
  		return err
  	}

  	return nil
  }
  ```
</CodeGroup>

## Multiple Progress Indicators

A job can have multiple independent progress indicators. This is useful when a job consists of multiple steps, that each benefits from having its own progress indicator.
To create a new progress indicator, call `context.progress(name)` with a unique `name` for the indicator.

<CodeGroup>
  ```python Python lines focus={5-6,15-18,27-28,33-34,38-39} theme={"system"}
  class MultiProgressWorkflowRoot(Task):
      n: int

      def execute(self, context: ExecutionContext) -> None:
          # initialize a progress indicator for the finalize task
          context.progress("finalize").add(1)

          process = context.submit_subtask(Process(self.n))
          context.submit_subtask(Cleanup(), depends_on=[process])

  class Process(Task):
      n: int

      def execute(self, context: ExecutionContext) -> None:
          # initialize two progress indicators for the two steps,
          # with a total work of n for each
          context.progress("step1").add(self.n)
          context.progress("step2").add(self.n)

          # now submit N subtasks for the two steps
          for _ in range(self.n):
              step1 = context.submit_subtask(Step1())
              context.submit_subtask(Step2(), depends_on=[step1])

  class Step1(Task):
      def execute(self, context: ExecutionContext) -> None:
          # 1 unit of work for step1 is done
          context.progress("step1").done(1)


  class Step2(Task):
      def execute(self, context: ExecutionContext) -> None:
          # 1 unit of work for step2 is done
          context.progress("step2").done(1)

  class Finalize(Task):
      def execute(self, context: ExecutionContext) -> None:
          # finalize is done
          context.progress("finalize").done(1)
  ```

  ```go Go lines expandable focus={7,32,37,62,74,86} theme={"system"}
  type MultiProgressWorkflowRoot struct {
      N int
  }

  func (t *MultiProgressWorkflowRoot) Execute(ctx context.Context) error {
      // initialize a progress indicator for the finalize task
      err := workflows.Progress("finalize").Add(ctx, 1)
      if err != nil {
          return err
      }

      process, err := workflows.SubmitSubtask(ctx, &Process{N: t.N})
      if err != nil {
          return err
      }

      _, err = workflows.SubmitSubtask(ctx, &Cleanup{}, subtask.WithDependencies(process))
      if err != nil {
          return err
      }

      return nil
  }

  type Process struct {
      N int
  }

  func (t *Process) Execute(ctx context.Context) error {
  	// initialize two progress indicators for the two steps,
  	// with a total work of n for each
      err := workflows.Progress("step1").Add(ctx, uint64(t.N))
      if err != nil {
          return err
      }

      err = workflows.Progress("step2").Add(ctx, uint64(t.N))
      if err != nil {
          return err
      }

      // now submit N subtasks for the two steps
      for range t.N {
          step1, err := workflows.SubmitSubtask(ctx, &Step1{})
          if err != nil {
              return err
          }

          _, err = workflows.SubmitSubtask(ctx, &Step2{}, subtask.WithDependencies(step1))
          if err != nil {
              return err
          }
      }

      return nil
  }

  type Step1 struct{}

  func (t *Step1) Execute(ctx context.Context) error {
      // 1 unit of work for step1 is done
      err := workflows.Progress("step1").Done(ctx, 1)
      if err != nil {
          return err
      }

      return nil
  }

  type Step2 struct{}

  func (t *Step2) Execute(ctx context.Context) error {
      // 1 unit of work for step2 is done
      err := workflows.Progress("step2").Done(ctx, 1)
      if err != nil {
          return err
      }

      return nil
  }

  type Cleanup struct{}

  func (t *Cleanup) Execute(ctx context.Context) error {
      // finalize is done
      err := workflows.Progress("finalize").Done(ctx, 1)
      if err != nil {
          return err
      }

      return nil
  }
  ```
</CodeGroup>

## Querying Progress

At any time during a job's execution, you can query the current progress of a job using the `find` method on the job client. The returned job object contains a `progress` field that contains the current progress of the job.

<CodeGroup>
  ```python Python theme={"system"}
  job = job_client.find(job_id)
  for indicator in job.progress:
      print(f"{indicator.label}: {indicator.done}/{indicator.total}")
  ```

  ```go Go theme={"system"}
  job, err := client.Jobs.Get(ctx, jobID)
  if err != nil {
      slog.Error("Failed to get job", slog.Any("error", err))
      return
  }

  for _, indicator := range job.Progress {
      fmt.Printf("%s: %d/%d\n", indicator.Label, indicator.Done, indicator.Total)
  }
  ```
</CodeGroup>

```plaintext Output theme={"system"}
finalize: 1/1
step1: 4/4
step2: 4/4
```

### Progress Display in interactive environments

When running in an interactive environment such as a Jupyter notebook and the cell output is a Tilebox job object, the
job is automatically rendered, including its progress indicators.

<CodeGroup>
  ```python Python theme={"system"}
  job = job_client.find(job_id)
  job  # trigger notebook cell output
  ```
</CodeGroup>

<Frame>
  <img alt="Tilebox Workflows job object as Jupyter Cell output" />

  <img alt="Tilebox Workflows job object as Jupyter Cell output" />
</Frame>

## Progress idempotency

Since tasks may fail and can subsequently be retried, it's possible that a task is executed more than once. This means that a task may report progress more than once.
To avoid double-counting such progress updates, Tilebox only considers the progress reported by the last execution of a task.


# Query telemetry
Source: https://docs.tilebox.com/workflows/run-and-inspect/query-telemetry

Query workflow logs and spans for a job from Python or Go.

Tilebox stores logs and spans for each workflow job. Use the jobs client to query that telemetry from notebooks, scripts, or automated diagnostics.

## Query job logs

`query_logs()` returns a `LogRecords` list. Pagination is handled automatically.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  job = client.jobs().find("019e07b1-916b-0630-f3ba-f1c33235d174")

  logs = client.jobs().query_logs(job)

  for record in logs:
      print(record.time, record.severity_text, record.body)
      print(record.attributes)
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"fmt"
  	"log/slog"
  	"time"

  	"github.com/google/uuid"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := workflows.NewClient()
  	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  	for record, err := range client.Jobs.QueryLogs(
  		ctx,
  		jobID,
  		workflows.WithSortDirection(workflows.Ascending),
  	) {
  		if err != nil {
  			slog.ErrorContext(ctx, "failed to query job logs", slog.Any("error", err))
  			return
  		}

  		fmt.Println(record.Time.Format(time.RFC3339Nano), record.Level, record.Body)
  		fmt.Println(record.Attributes)
  	}
  }
  ```
</CodeGroup>

Each log record includes:

* `time`
* `severity_number` and `severity_text`
* `body`
* `trace_id` and `span_id`
* `attributes`
* `runner_attributes`

### As pandas DataFrame

Use `to_pandas()` to convert log records to a pandas DataFrame.

```python Python theme={"system"}
logs_df = client.jobs().query_logs(job).to_pandas()
logs_df[["time", "severity_text", "body"]]
```

<Frame>
  <img alt="Job Logs as Pandas DataFrame" />

  <img alt="Job Logs as Pandas DataFrame" />
</Frame>

## Query job spans

`query_spans()` returns a `Spans` list. Pagination is handled automatically.

<CodeGroup>
  ```python Python theme={"system"}
  spans = client.jobs().query_spans(job.id)

  for span in spans:
      print(span.name, span.status_code, span.duration)
      print(span.attributes)
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"fmt"
  	"log/slog"

  	"github.com/google/uuid"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := workflows.NewClient()
  	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  	for span, err := range client.Jobs.QuerySpans(
  		ctx,
  		jobID,
  		workflows.WithSortDirection(workflows.Ascending),
  	) {
  		if err != nil {
  			slog.ErrorContext(ctx, "failed to query job spans", slog.Any("error", err))
  			return
  		}

  		fmt.Println(span.Name, span.StatusCode, span.Duration())
  		fmt.Println(span.Attributes)
  	}
  }
  ```
</CodeGroup>

Each span includes:

* `start_time` and `end_time`
* `duration`
* `trace_id`, `span_id`, and `parent_span_id`
* `name`
* `status_code` and `status_message`
* `attributes`
* `runner_attributes`
* `events`

### As pandas DataFrame

Use `to_pandas()` to convert spans to a pandas DataFrame.

```python Python theme={"system"}
spans_df = client.jobs().query_spans(job).to_pandas()

slow_spans = spans_df.sort_values("duration", ascending=False).head(10)
slow_spans[["name", "duration", "start_time"]]
```

<Frame>
  <img alt="Job Traces as Pandas DataFrame" />

  <img alt="Job Traces as Pandas DataFrame" />
</Frame>

Nested `attributes`, `runner_attributes`, and `events` stay as Python objects in DataFrame columns. Span DataFrames include a computed `duration` column.


# Tracing
Source: https://docs.tilebox.com/workflows/run-and-inspect/tracing

Use built-in workflow traces and custom spans to inspect job execution, task duration, and bottlenecks.

Tilebox traces workflow jobs automatically. Job submission creates a root trace, runners continue that trace across machines, and every task execution creates a span.

<Frame>
  <img alt="Job Execution Trace View" />

  <img alt="Job Execution Trace View" />
</Frame>

Built-in traces connect task order, dependencies, parallel execution, task duration, task status, runner identity, service identity, and logs emitted while a span was active.

## Add custom spans

Use `context.tracer` inside a task to add spans around meaningful parts of your own code.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import ExecutionContext, Task

  class ProcessScene(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          with context.tracer.span("download-scene") as span:
              span.set_attribute("scene_id", self.scene_id)
              # download input data

          with context.tracer.span("compute-index"):
              # perform expensive computation
              pass
  ```

  ```go Go theme={"system"}
  package tasks

  import (
  	"context"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	return workflows.WithSpan(ctx, "compute-index", func(ctx context.Context) error {
  		// perform expensive computation
  		return nil
  	})
  }
  ```
</CodeGroup>

Custom spans are nested under the current task span. Logs emitted inside the span are correlated with its `trace_id` and `span_id`.

## Span status and exceptions

If a task raises an exception, Tilebox records the exception on the task span and marks the span as failed before the task is retried or marked failed.

For finer-grained error reporting, record errors on your custom spans before re-raising them.

<CodeGroup>
  ```python Python theme={"system"}
  class ProcessScene(Task):
      scene_id: str

      def execute(self, context: ExecutionContext) -> None:
          with context.tracer.span("publish-output") as span:
              try:
                  # publish output
                  pass
              except Exception as error:
                  span.record_exception(error)
                  raise
  ```

  ```go Go theme={"system"}
  package tasks

  import (
  	"context"
  	"fmt"

  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  type ProcessScene struct{}

  func (t *ProcessScene) Execute(ctx context.Context) error {
  	return workflows.WithSpan(ctx, "publish-output", func(ctx context.Context) error {
  		if err := publishOutput(); err != nil {
  			return fmt.Errorf("failed to publish output: %w", err)
  		}
  		return nil
  	})
  }

  func publishOutput() error {
  	return nil
  }
  ```
</CodeGroup>

## Query spans

You can retrieve spans for a job through the jobs client. Python results can also be converted to a pandas DataFrame.

<CodeGroup>
  ```python Python theme={"system"}
  from tilebox.workflows import Client

  client = Client()
  job = client.jobs().submit("process-scene", ProcessScene(scene_id="S2A_001"))

  spans = client.jobs().query_spans(job)
  for span in spans:
      print(span.name, span.status_code, span.duration)

  df = spans.to_pandas()
  ```

  ```go Go theme={"system"}
  package main

  import (
  	"context"
  	"fmt"
  	"log/slog"
  	"time"

  	"github.com/google/uuid"
  	"github.com/tilebox/tilebox-go/workflows/v1"
  )

  func main() {
  	ctx := context.Background()
  	client := workflows.NewClient()
  	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

  	for span, err := range client.Jobs.QuerySpans(ctx, jobID) {
  		if err != nil {
  			slog.ErrorContext(ctx, "failed to query job spans", slog.Any("error", err))
  			return
  		}

  		fmt.Printf("%s %-40s %s\n",
  			span.StartTime.Format(time.RFC3339Nano),
  			span.Name,
  			span.Duration(),
  		)
  	}
  }
  ```
</CodeGroup>

See [Query telemetry](/workflows/run-and-inspect/query-telemetry) for the log and span query APIs.

## Export to another backend

Tilebox stores traces by default. To export spans to your own observability backend as well, configure an [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) integration when the runner process starts.