Tilebox uses Protocol Buffers, with a custom generation tool, combined with standard Go data structures.

Protocol Buffers (often referred to as protobuf) is a schema definition language with an efficient binary format and native language support for lots of languages, including Go. Protocol buffers are open source since 2008 and are maintained by Google.

​
tilebox-generate

Protobuf schemas are typically defined in a .proto file, and then converted to a native Go struct using the protobuf compiler. Tilebox datasets already define a protobuf schema as well, and automate the generation of Go structs for existing datasets through a quick tilebox-generate command-line tool.

See Installation for more details on how to install tilebox-generate.

tilebox-generate --dataset open_data.copernicus.sentinel1_sar

The preceding command will generate a ./protogen/tilebox/v1/sentinel1_sar.pb.go file. More flags can be set to change the default output folders, package name, etc.

This file contains everything needed to work with the Sentinel-1 SAR dataset. It’s recommended to check the generated files you use in your version control system.

If you open this file, you will see that it starts with // Code generated by protoc-gen-go. DO NOT EDIT.. It means that the file was generated by the protoc-gen-go tool, which is part of the protobuf compiler. After editing a dataset, you can call the generate command again to ensure that the changes are reflected in the generated file.

The file contains a Sentinel1Sar struct, which is a Go struct that represents a datapoint in the dataset.

Go
type Sentinel1Sar struct {
  xxx_hidden_GranuleName         *string                `protobuf:"bytes,1,opt,name=granule_name,json=granuleName"`
  xxx_hidden_ProcessingLevel     v1.ProcessingLevel     `protobuf:"varint,2,opt,name=processing_level,json=processingLevel,enum=datasets.v1.ProcessingLevel"`
  // more fields
}

Notice that the fields are private (starting with a lowercase letter), so they are not accessible. Protobuf hides the fields and provides getters and setters to access them.

​
Protobuf 101

​
Initializing a message

Here is how to initialize a v1.Sentinel1Sar message.

Go
import (
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"
)

datapoint := v1.Sentinel1Sar_builder{
	Time:        timestamppb.New(time.Now()),
	GranuleName: proto.String("S1A_EW_GRDH_1SSH_20141004T020507_20141004T020611_002673_002FAF_8645_COG.SAFE"),
	ProductType: proto.String("EW_GRDH_1S-COG"),
	FileSize:    proto.Int64(488383473),
}.Build()

Protobuf fields are private and provides a builder pattern to create a message.

proto.String is a helper function that converts string to *string. This allows protobuf to differentiate between a field that is set to an empty string and a field that is not set (nil). An exhaustive list of those helper functions can be found here.

Only primitives have a proto.XXX helper function. Complex types such as timestamps, durations, UUIDs, and geometries have a constructor function.

​
Getters and setters

Protobuf provides methods to get, set, clear and check if a field is set.

Go
fmt.Println(datapoint.GetGranuleName())

datapoint.SetGranuleName("my amazing granule")

datapoint.ClearGranuleName()

if datapoint.HasGranuleName() {
  fmt.Println("Granule name is set")
}

Getters for primitive types will return a Go native type (for example, int64, string, etc.). Getters for complex types such as timestamps, durations, UUIDs, and geometries can also be converted to more standard types using AsXXX methods.

​
Well known types

Beside Go primitives, Tilebox supports some well known types:

They have a couple of useful methods to work with them.

​
Constructors

Go
import (
  "github.com/paulmach/orb"
	datasetsv1 "github.com/tilebox/tilebox-go/protogen/go/datasets/v1"
	"google.golang.org/protobuf/types/known/durationpb"
	"google.golang.org/protobuf/types/known/timestamppb"
)

timestamppb.New(time.Now())
durationpb.New(10 * time.Second)
datasetsv1.NewUUID(uuid.New())
datasetsv1.NewGeometry(orb.Point{1, 2})

​
CheckValid method

CheckValid returns an error if the field is invalid.

Go
err := datapoint.GetTime().CheckValid()
if err != nil {
  fmt.Println(err)
}

​
IsValid method

IsValid reports whether the field is valid. It’s equivalent to CheckValid == nil.

Go
if datapoint.GetTime().IsValid() {
  fmt.Println("Valid")
}

​
AsXXX methods

AsXXX methods convert the field to a more user friendly type.

  • AsUUID will convert a datasetsv1.UUID field to a uuid.UUID type
  • AsTime will convert a timestamppb.Timestamp field to a time.Time type
  • AsDuration will convert a durationpb.Duration field to a time.Duration type
  • AsGeometry will convert a datasetsv1.Geometry field to an orb.Geometry interface
Go
datapoint.GetId().AsUUID() // uuid.UUID
datapoint.GetTime().AsTime() // time.Time
datapoint.GetDuration().AsDuration() // time.Duration
datapoint.GetGeometry().AsGeometry() // orb.Geometry

Those methods performs conversion on a best-effort basis. Type validity must be checked beforehand using IsValid or CheckValid methods.

​
Common data operations

Datapoints are contained in a standard Go slice so all the usual slice operations and slice functions can be used.

The usual pattern to iterate over data in Go is by using a for loop.

As an example, here is how to extract the copernicus_id fields from the datapoints.

Go
// assuming datapoints has been filled using `client.Datapoints.QueryInto` method
var datapoints []*v1.Sentinel1Sar

copernicusIDs := make([]uuid.UUID, len(datapoints))
for i, dp := range datapoints {
  copernicusIDs[i] = dp.GetCopernicusId().AsUUID()
}

Here is an example of filtering out datapoints that have been published before January 2000 and are not from the Sentinel-1C platform.

Go
jan2000 := time.Date(2000, time.January, 1, 0, 0, 0, 0, time.UTC)
// slice of length of 0, but preallocate a capacity of len(datapoints)
s1cDatapoints := make([]*v1.Sentinel1Sar, 0, len(datapoints))

for _, dp := range datapoints {
  if dp.GetPublished().AsTime().Before(jan2000) {
    continue
  }
  if dp.GetPlatform() != "S1C" {
    continue
  }

  s1cDatapoints = append(s1cDatapoints, proto.CloneOf(dp)) // Copy the underlying data
}

​
Converting to JSON

Protobuf messages can be converted to JSON without loss of information. This is useful for interoperability with other systems that doesn’t use protobuf. A guide on protoJSON can be found format here: https://protobuf.dev/programming-guides/json/

Go
originalDatapoint := datapoints[0]

// Convert proto.Message to JSON as bytes
jsonDatapoint, err := protojson.Marshal(originalDatapoint)
if err != nil {
  log.Fatalf("Failed to marshal datapoint: %v", err)
}
fmt.Println(string(jsonDatapoint))
Output
{"time":"2001-01-01T00:00:00Z","id":{"uuid":"AOPHpzQAAmV2MZ4+Zv+JGg=="},"ingestionTime":"2025-03-25T10:26:10.577385176Z","granuleName":"MCD12Q1.A2001001.h02v08.061.2022146033342.hdf","geometry":{"wkb":"AQMAAAABAAAABQAAAFIi9vf7TmTAXsX3////I0Bexff///9jwAAAAAAAAAAACUn4//+/YsAAAAAAAAAAAC7AdjgMCmPAXsX3////I0BSIvb3+05kwF7F9////yNA"},"endTime":"2001-12-31T23:59:59Z","horizontalTileNumber":"2","verticalTileNumber":"8","tileId":"51002008","fileSize":"176215","checksum":"771212892","checksumType":"CKSUM","dayNightFlag":"Day","publishedAt":"2022-06-23T10:58:13.895Z"}

It can also be converted back to a proto.Message.

Go
// Convert JSON bytes to proto.Message
unmarshalledDatapoint := &v1.Sentinel1Sar{}
err = protojson.Unmarshal(jsonDatapoint, unmarshalledDatapoint)
if err != nil {
  log.Fatalf("Failed to unmarshal datapoint: %v", err)
}

fmt.Println("Are both equal?", proto.Equal(unmarshalledDatapoint, originalDatapoint))
Output
Are both equal? true