Datasets

Build a spatio-temporal catalog

Learn how to create a custom dataset catalog with the Python SDK.

Ingest into a spatio-temporal catalog

Learn how to ingest GeoParquet metadata into an existing spatio-temporal catalog.

Dataset types

Each dataset is of a specific type. Each dataset type comes with a set of required fields for each data point. The dataset type also determines the query capabilities for a dataset, for example, whether a dataset supports time-based queries or additionally also spatially filtered queries. To find out which fields are required for each dataset type check out the documentation for the available dataset types below.

Timeseries Data

Each data point is linked to a specific point in time. Common for satellite telemetry, or other time-based data. Supports efficient time-based queries.

Spatio-temporal Data

Each data point is linked to a specific point in time and a location on the Earth’s surface. Common for satellite imagery. Supports efficient time-based and spatially filtered queries.

Dataset specific fields

Additionally, each dataset has a set of fields that are specific to that dataset. Fields are defined during dataset creation. That way, all data points in a dataset are strongly typed and are validated during ingestion. The required fields of the dataset type, as well as the custom fields specific to each dataset together make up the dataset schema. Once a dataset schema is defined, existing fields cannot be removed or edited as soon as data has been ingested into it. You can add new non-queryable fields to a non-empty dataset because all custom fields are optional. You cannot add a new queryable field or change whether an existing field is queryable after ingesting data.

Empty datasets are the exception. If all collections are empty, you can freely edit the data schema and its queryable field annotations because no existing datapoints can conflict with the change.

Field types

When defining the data schema, you can specify each field’s type. The following field types are supported.

Primitives

Type	Description	Example value
string	A string of characters of arbitrary length.	`Some string`
int32	A 32-bit signed integer.	`123`
int64	A 64-bit signed integer.	`123`
uint64	A 64-bit unsigned integer.	`123`
float64	A 64-bit floating-point number.	`123.45`
bool	A boolean.	`true`
bytes	A sequence of arbitrary length bytes.	`0xAF1E28D4`

Time

Type	Description	Example value
Duration	A signed, fixed-length span of time represented as a count of seconds and fractions of seconds at nanosecond resolution. See Duration for more information.	`12s 345ms`
Timestamp	A point in time, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time. See Timestamp for more information.	`2023-05-17T14:30:00Z`

Identifier

Type	Description	Example value
UUID	A universally unique identifier (UUID).	`126a2531-c98d-4e06-815a-34bc5b1228cc`

Geospatial

Type	Description	Example value
Geometry	Geospatial geometries of type Point, LineString, Polygon or MultiPolygon.	`POLYGON ((12.3 -5.4, 12.5 -5.4, ...))`

STAC metadata

Structured STAC types preserve common metadata without flattening it into separate custom fields. In Python, import these types from tilebox.datasets.schema.

Type	Description	Example contents
Assets	Files associated with a datapoint, based on the STAC Asset Object.	Image bands and thumbnails
Authentication	Authentication schemes described by the STAC Authentication Extension.	S3 credentials
Links	Relationships to other resources, based on the STAC Link Object.	Canonical and related links
ProcessingSoftware	Software and versions described by the STAC Processing Extension.	Processor name and version
Provider	Organizations that produced, processed, hosted, or licensed the data.	Producer and host details
Storage	Storage-system details described by the STAC Storage Extension.	S3 region and requester-pays settings

See Assets and storage for the asset access model.

Arrays

Every type is also available as an array, allowing to ingest multiple values of the underlying type for each data point. The size of the array is flexible, and can be different for each data point.

Queryable fields

Custom fields can be marked as queryable when you define the dataset schema. Tilebox projects these fields into query storage so you can filter datapoints by their values together with temporal, spatial, and collection filters.

Constraint	Limit
Supported types	`string`, `bool`, `int32`, `int64`, `uint64`, and `float64`
Repeated fields	Not supported
Queryable string fields	At most two per dataset
Queryable string values	At most 1,024 Unicode code points
Field location	Top-level custom fields only
Field names	Snake case, starting with a letter, and at most 100 characters

Select queryable fields before ingesting datapoints. For a non-empty dataset, you cannot change whether an existing field is queryable or add a new queryable field.

Listing datasets

You can use your client instance to access the datasets available to you. To list all available datasets, use the datasets method of the client.

from tilebox.datasets import Client

client = Client()
datasets = client.datasets()
print(datasets)

package main

import (
  "context"
  "log"

  "github.com/tilebox/tilebox-go/datasets/v1"
)

func main() {
  ctx := context.Background()

  client := datasets.NewClient()

  allDatasets, err := client.Datasets.List(ctx)
  if err != nil {
    log.Fatalf("Failed to list datasets: %v", err)
  }

  for _, dataset := range allDatasets {
    log.Println(dataset)
  }
}

Output

open_data:
    asf:
        ers_sar: European Remote Sensing Satellite (ERS) Synthetic Aperture Radar (SAR) Granules
    copernicus:
        landsat8_oli_tirs: Landsat-8 is part of the long-running Landsat programme ...
        sentinel1_sar: The Sentinel-1 mission is the European Radar Observatory for the ...
        sentinel2_msi: Sentinel-2 is equipped with an optical instrument payload that samples ...
        sentinel3_olci: OLCI (Ocean and Land Colour Instrument) is an optical instrument used to ...
        ...

Once you have your dataset object, you can use it to list the available collections for the dataset.

In python, if you’re using an IDE or an interactive environment with auto-complete, you can use it on your client instance to discover the datasets available to you. Type client. and trigger auto-complete after the dot to do so.

Creating / Updating a dataset

Create a dataset by defining its kind, custom fields, and field annotations. The Tilebox Console provides a visual schema editor.

Python
Go
CLI

from datetime import timedelta
from tilebox.datasets import Client
from tilebox.datasets.data.datasets import DatasetKind

client = Client()

dataset = client.create_or_update_dataset(
    DatasetKind.SPATIOTEMPORAL,
    "my_landsat8_oli_tirs",
    [
        {
            "name": "granule_name",
            "type": str,
            "description": "The name of the Landsat-8 granule.",
            "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1",
            "queryable": True,
        },
        {
            "name": "cloud_cover",
            "type": float,
            "description": "Cloud cover percentage",
            "example_value": "91",
            "queryable": True,
        },
        {
            "name": "proj_shape",
            "type": list[int],
            "description": "Raster shape",
            "example_value": "[6971, 7801]",
        },
    ],
    name="Personal Landsat-8 catalog",
)

dataset, err := client.Datasets.Create(ctx,
  datasets.KindSpatiotemporal,
  "my_landsat8_oli_tirs",
  "Personal Landsat-8 catalog",
  []datasets.Field{
    field.String("granule_name").Description("The name of the Earth View Granule").ExampleValue("20220830_185202_SN18_10N_552149_5297327").Queryable(),
    field.Float64("cloud_cover").Description("Cloud cover percentage").ExampleValue("91").Queryable(),
    field.Int64("proj_shape").Repeated().Description("Raster shape").ExampleValue("[6971, 7801]"),
  }
)

schema.json

{
  "kind": "spatiotemporal",
  "fields": [
    {
      "name": "granule_name",
      "type": "string",
      "description": "The name of the Landsat-8 granule.",
      "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1",
      "queryable": true
    },
    {
      "name": "cloud_cover",
      "type": "float64",
      "description": "Cloud cover percentage",
      "example_value": "91",
      "queryable": true
    },
    {
      "name": "proj_shape",
      "type": "int64",
      "repeated": true,
      "description": "Raster shape",
      "example_value": "[6971, 7801]"
    }
  ]
}

tilebox dataset create \
  --name "Personal Landsat-8 catalog" \
  --code-name my_landsat8_oli_tirs \
  --schema-file schema.json

This code will create a new catalog with 7 fields in total. 4 of those fields are auto-generated by choosing the spatio-temporal dataset type, and 3 (granule_name, cloud_cover, proj_shape) are custom fields that are defined. granule_name and cloud_cover can be used in custom field filters; the repeated proj_shape field cannot be queryable.

If a dataset with the same code_name already exists, it will be updated instead.

Accessing a dataset

Each dataset has an automatically generated slug that can be used to access it. The slug is the name of the group, followed by a dot, followed by the dataset code name. For example, the slug for the Sentinel-2 MSI dataset, which is part of the open_data.copernicus group, is open_data.copernicus.sentinel2_msi. To access a dataset, use the dataset method of your client instance and pass the slug of the dataset as an argument.

from tilebox.datasets import Client

client = Client()
s2_msi_dataset = client.dataset("open_data.copernicus.sentinel2_msi")

s2MsiDataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")

Once you have your dataset object, you can use it to access available collections for the dataset.

Deleting a dataset

Datasets can be deleted through the Tilebox Console by clicking the Delete button in the dataset page. Empty datasets will be deleted right away. A dataset is considered empty if none of its collections contain any data points. A non-empty dataset can also be deleted, but is a destructive operation. Every data point in every collection of the dataset will be deleted. As a safety measure, Tilebox soft-deletes the dataset for 7 days before permanently deleting it. Please get in touch if you deleted a dataset by accident and want to restore it.

Get Started

Agents and AI Tools

Workflows

Datasets

Build a spatio-temporal catalog

Ingest into a spatio-temporal catalog

Dataset types

Timeseries Data

Spatio-temporal Data

Dataset specific fields

Field types

Primitives

Time

Identifier

Geospatial

STAC metadata

Arrays

Queryable fields

Listing datasets

Creating / Updating a dataset

Accessing a dataset

Deleting a dataset

​Related Guides

Build a spatio-temporal catalog

Ingest into a spatio-temporal catalog

​Dataset types

Timeseries Data

Spatio-temporal Data

​Dataset specific fields

​Field types

​Primitives

​Time

​Identifier

​Geospatial

​STAC metadata

​Arrays

​Queryable fields

​Listing datasets

​Creating / Updating a dataset

​Accessing a dataset

​Deleting a dataset

Related Guides

Dataset types

Dataset specific fields

Field types

Primitives

Time

Identifier

Geospatial

STAC metadata

Arrays

Queryable fields

Listing datasets

Creating / Updating a dataset

Accessing a dataset

Deleting a dataset