DataIngestion

Ingestion

Turn source data into validated datasources.
View as Markdown

Ingestion is the stage that reads source files, applies a schema, and creates a datasource that can be used for onboarding.

It connects the data you prepared in storage with the schema that describes it. The result is a datasource: a validated, platform-managed representation of the source data.

How ingestion works

1

Storage

Source files are read from a connected storage location.

2

Schema

The schema defines how each column should be interpreted during ingestion.

3

Configuration

The ingestion configuration combines the schema with ingestion-specific settings.

4

Job

The ingestion job runs against a storage path using a processing compute configuration.

5

Datasource

The completed job creates or updates the datasource used by onboarding.

Data ingestion flow.

Initial and incremental ingestion

The first ingestion job for a source creates a datasource.

Later ingestion jobs can update that datasource with additional data. Updates point to the existing datasource id and can include an increment_label to identify the new increment.