DataIngestion

Configuration

Create reusable ingestion settings for datasource jobs.
View as Markdown

An ingestion configuration defines how source data is interpreted during ingestion.

It references the data schema used to validate the source columns. Ingestion jobs then reuse the configuration when creating or updating datasources.

Create a configuration

Create an ingestion configuration after you have created or selected a data schema.

POST
/data-processing-ingestion-configurations
1curl -X POST https://api.hi.umnai.com/data-processing-ingestion-configurations \
2     -H "Authorization: Bearer <token>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "name": "Sample Ingestion Configuration",
6  "data_schema_id": "c1342ae9bd985c0d9c5fa5222609c3fa"
7}'
Response
1{
2  "id": "d2f74730b1467213748b3750b3ad87ac",
3  "name": "Sample Ingestion Configuration",
4  "account": {
5    "id": "e268443e43d93dab7ebef303bbe9642f",
6    "name": "Sample Account"
7  },
8  "created_at": "2026-03-15T10:15:30.000000Z",
9  "observation_specification": {
10    "observation_type": "observation_type",
11    "observation_source": "observation_source",
12    "observation_shard_id": "observation_shard_id",
13    "observation_file_index": "observation_file_index",
14    "observation_index": "observation_index",
15    "observation_mask_index": "observation_mask_index",
16    "observation_mask_file_index": "observation_mask_file_index",
17    "observation_increment_index": "observation_increment_index"
18  },
19  "created_by": {
20    "id": "dad46b2058752ffde5eaf02f1eb6abd5",
21    "name": "Sample User"
22  },
23  "updated_at": "2026-03-15T10:15:30.000000Z",
24  "data_schema": {
25    "id": "c1342ae9bd985c0d9c5fa5222609c3fa",
26    "name": "Sample Data Schema"
27  }
28}

Save the returned configuration id. You will use it when starting ingestion jobs.

Observation fields

During ingestion, the platform can add observation fields that help identify, track, and process rows.

By default, these fields use standard names such as observation_type, observation_source, and observation_increment_index. You only need to configure observation_specification when your source data already uses different column names for these concepts.

Training and test rows

If your source data includes a column that separates training rows from test rows, map that column through observation_specification.observation_type.

If no observation type is provided, ingested rows are treated as training data.

Retrieve configurations

Retrieve ingestion configurations when you need to reuse an existing configuration.

GET
/data-processing-ingestion-configurations
1curl https://api.hi.umnai.com/data-processing-ingestion-configurations \
2     -H "Authorization: Bearer <token>"
Response
1{
2  "data": [
3    {
4      "id": "ea8b745ead39ec442fc3b4fba359bf98",
5      "name": "Sample Organisation",
6      "created_at": "2026-03-11T16:04:19.749000Z",
7      "updated_at": null,
8      "status": "ACTIVE"
9    }
10  ],
11  "pagination": {
12    "total_items": 1,
13    "items_per_page": 10,
14    "current_page": 1,
15    "total_pages": 1
16  }
17}

To inspect a specific configuration, retrieve it by id.

GET
/data-processing-ingestion-configurations/:dataProcessingIngestionConfigurationId
1curl https://api.hi.umnai.com/data-processing-ingestion-configurations/{dataProcessingIngestionConfigurationId} \
2     -H "Authorization: Bearer <token>"
Response
1{
2  "data": [
3    {
4      "id": "ea8b745ead39ec442fc3b4fba359bf98",
5      "name": "Sample Organisation",
6      "status": "ACTIVE",
7      "created_at": "2026-03-11T16:04:19.749000Z",
8      "updated_at": null
9    }
10  ],
11  "pagination": {
12    "total_items": 1,
13    "items_per_page": 10,
14    "current_page": 1,
15    "total_pages": 1
16  }
17}

Update a configuration

Update an ingestion configuration when you need to rename it or point it to a different data schema.

PATCH
/data-processing-ingestion-configurations/:dataProcessingIngestionConfigurationId
1curl -X PATCH https://api.hi.umnai.com/data-processing-ingestion-configurations/dataProcessingIngestionConfigurationId \
2     -H "Authorization: Bearer <token>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "name": "Sample Ingestion Configuration"
6}'
Response
1{
2  "id": "d2f74730b1467213748b3750b3ad87ac",
3  "name": "Sample Ingestion Configuration",
4  "account": {
5    "id": "e268443e43d93dab7ebef303bbe9642f",
6    "name": "Sample Account"
7  },
8  "created_at": "2026-03-15T10:15:30.000000Z",
9  "observation_specification": {
10    "observation_type": "observation_type",
11    "observation_source": "observation_source",
12    "observation_shard_id": "9d1d3ae7d37a564b88b9eef14e005c2b",
13    "observation_file_index": "observation_file_index",
14    "observation_index": "observation_index",
15    "observation_mask_index": "observation_mask_index",
16    "observation_mask_file_index": "observation_mask_file_index",
17    "observation_increment_index": "observation_increment_index"
18  },
19  "created_by": {
20    "id": "dad46b2058752ffde5eaf02f1eb6abd5",
21    "name": "Sample User"
22  },
23  "updated_at": "2026-03-15T10:15:30.000000Z",
24  "data_schema": {
25    "id": "c1342ae9bd985c0d9c5fa5222609c3fa",
26    "name": "Sample Data Schema"
27  }
28}

Schema changes apply to future ingestion jobs that use the configuration. Existing datasources keep the schema that was applied when they were ingested.

Delete a configuration

Delete an ingestion configuration when it should no longer be used by future ingestion jobs.

DELETE
/data-processing-ingestion-configurations/:dataProcessingIngestionConfigurationId
1curl -X DELETE https://api.hi.umnai.com/data-processing-ingestion-configurations/dataProcessingIngestionConfigurationId \
2     -H "Authorization: Bearer <token>"