Data

Data

Prepare source data and turn it into model-ready datasets.

View as Markdown

Data workflows in the platform turn source files into datasets that can be used for model training. The flow has two main stages:

  • Ingestion validates source data and creates a datasource.
  • Onboarding turns that datasource into a dataset ready for training.

Source storage and account storage have different roles and permissions. They can use the same S3 bucket, but dedicated buckets are strongly recommended. See Account storage for details.

How data moves through the platform

1

Prepare your data

Organise the source files you want to use for ingestion.

2

Connect source storage

Give the platform read-only access to the storage location that contains your source data. See Source storage.

3

Define the schema

Describe how each source column should be interpreted, including field types, designations, and encoding options.

4

Ingest the data

Run ingestion to validate the source data and create or update a datasource.

5

Onboard the dataset

Select features, define the target, and prepare the datasource for model training.