DataOnboarding

Jobs

Run onboarding to create or update datasets.
View as Markdown

An onboarding job runs an onboarding configuration against a datasource.

Jobs are asynchronous. Create a job, then retrieve it to monitor status, progress, current stage, and the dataset produced by the run.

Create an initial job

An initial onboarding job creates a new dataset.

POST
/data-processing-onboarding-jobs
1curl -X POST https://api.hi.umnai.com/data-processing-onboarding-jobs \
2     -H "Authorization: Bearer <token>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "job_type": "INITIAL",
6  "data_processing_configuration_id": "3dd973d3fda8604fba040381bfcaab29",
7  "dataset_name": "Sample Dataset",
8  "datasource_id": "1ea7e575defdf6bc3f26a3f127e98170",
9  "name": "Sample Initial Onboarding Job",
10  "compute_configuration_id": "351df95006b1350e597762a1d58e9319"
11}'
Response
1{
2  "job_type": "INITIAL",
3  "data_processing_configuration": {
4    "id": "3dd973d3fda8604fba040381bfcaab29",
5    "name": "Sample Onboarding Configuration"
6  },
7  "datasource": {
8    "id": "1ea7e575defdf6bc3f26a3f127e98170",
9    "name": "Sample Datasource"
10  },
11  "dataset": {
12    "id": "3c4d09e4ef50b370ae0efacdb43ec2dd",
13    "name": "Sample Dataset"
14  },
15  "id": "7c2e73157a35658c245511bcf2a95195",
16  "name": "Sample Initial Onboarding Job",
17  "account": {
18    "id": "e268443e43d93dab7ebef303bbe9642f",
19    "name": "Sample Account"
20  },
21  "compute_configuration": {
22    "id": "351df95006b1350e597762a1d58e9319",
23    "name": "Sample Compute Configuration"
24  },
25  "status": "STARTING",
26  "maximum_runtime_seconds": 3600,
27  "created_at": "2026-03-15T10:15:30.000000Z",
28  "created_by": {
29    "id": "dad46b2058752ffde5eaf02f1eb6abd5",
30    "name": "Sample User"
31  },
32  "updated_at": "2026-03-15T10:15:30.000000Z",
33  "stage": "initializing",
34  "progress": 0,
35  "stage_progress": 0
36}

The job references the datasource, onboarding configuration, and processing compute configuration. It also names the dataset that will be created.

Create an incremental job

An incremental onboarding job adds new data to an existing dataset.

POST
/data-processing-onboarding-jobs
1curl -X POST https://api.hi.umnai.com/data-processing-onboarding-jobs \
2     -H "Authorization: Bearer <token>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "job_type": "INCREMENT",
6  "data_processing_configuration_id": "3dd973d3fda8604fba040381bfcaab29",
7  "datasource_id": "3914cecdf57d2eade4435a94f82faf4e",
8  "dataset_id": "3c4d09e4ef50b370ae0efacdb43ec2dd",
9  "increment_label": "March update",
10  "name": "Sample Incremental Onboarding Job",
11  "compute_configuration_id": "351df95006b1350e597762a1d58e9319"
12}'
Response
1{
2  "job_type": "INCREMENT",
3  "data_processing_configuration": {
4    "id": "3dd973d3fda8604fba040381bfcaab29",
5    "name": "Sample Onboarding Configuration"
6  },
7  "datasource": {
8    "id": "3914cecdf57d2eade4435a94f82faf4e",
9    "name": "Sample Incremental Datasource"
10  },
11  "dataset": {
12    "id": "3c4d09e4ef50b370ae0efacdb43ec2dd",
13    "name": "Sample Dataset"
14  },
15  "increment_label": "March update",
16  "id": "8b1130720f054149ac83bca083f8d216",
17  "name": "Sample Incremental Onboarding Job",
18  "account": {
19    "id": "e268443e43d93dab7ebef303bbe9642f",
20    "name": "Sample Account"
21  },
22  "compute_configuration": {
23    "id": "351df95006b1350e597762a1d58e9319",
24    "name": "Sample Compute Configuration"
25  },
26  "status": "STARTING",
27  "maximum_runtime_seconds": 3600,
28  "created_at": "2026-03-16T10:15:30.000000Z",
29  "created_by": {
30    "id": "dad46b2058752ffde5eaf02f1eb6abd5",
31    "name": "Sample User"
32  },
33  "updated_at": "2026-03-16T10:15:30.000000Z",
34  "stage": "initializing",
35  "progress": 0,
36  "stage_progress": 0
37}

Incremental jobs use a newer datasource and can include an increment_label to identify the update.

Monitor jobs

Retrieve jobs to monitor onboarding progress or inspect previous runs.

GET
/data-processing-onboarding-jobs
1curl https://api.hi.umnai.com/data-processing-onboarding-jobs \
2     -H "Authorization: Bearer <token>"
Response
1{
2  "data": [
3    {
4      "id": "ea8b745ead39ec442fc3b4fba359bf98",
5      "name": "Sample Organisation",
6      "status": "ACTIVE",
7      "created_at": "2026-03-11T16:04:19.749000Z",
8      "updated_at": null
9    }
10  ],
11  "pagination": {
12    "total_items": 1,
13    "items_per_page": 10,
14    "current_page": 1,
15    "total_pages": 1
16  }
17}

To inspect a specific job, retrieve it by id.

GET
/data-processing-onboarding-jobs/:dataProcessingOnboardingJobId
1curl https://api.hi.umnai.com/data-processing-onboarding-jobs/{dataProcessingOnboardingJobId} \
2     -H "Authorization: Bearer <token>"
Response
1{
2  "data": [
3    {
4      "id": "ea8b745ead39ec442fc3b4fba359bf98",
5      "name": "Sample Organisation",
6      "status": "ACTIVE",
7      "created_at": "2026-03-11T16:04:19.749000Z",
8      "updated_at": null
9    }
10  ],
11  "pagination": {
12    "total_items": 1,
13    "items_per_page": 10,
14    "current_page": 1,
15    "total_pages": 1
16  }
17}

Job responses include the job status, overall progress, current stage, and stage progress.

StatusMeaning
STARTINGThe job has been created and is being prepared.
RUNNINGThe job is processing data.
FINALIZINGThe job is completing final processing steps.
COMPLETEDThe job finished successfully.
FAILEDThe job stopped because of an error.
STOPPEDThe job was stopped before completion.

Initial job stages

An initial onboarding job can move through these stages:

  • Data validation
  • Masking
  • Analysis
  • Statistics
  • Preprocessing
  • Sharding
  • Transformation
  • Initialisation
  • Persistence
  • Teardown

Incremental job stages

An incremental onboarding job can move through these stages:

  • Masking
  • Analysis
  • Statistics
  • Preprocessing
  • Sharding
  • Transformation
  • Metadata
  • Persistence
  • Teardown

The masking stage is only used when unseen category handling is enabled.

Update a job

Update an onboarding job when you need to rename it.

PATCH
/data-processing-onboarding-jobs/:dataProcessingOnboardingJobId
1curl -X PATCH https://api.hi.umnai.com/data-processing-onboarding-jobs/dataProcessingOnboardingJobId \
2     -H "Authorization: Bearer <token>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "name": "New Sample Onboarding Job"
6}'
Response
1{
2  "job_type": "INITIAL",
3  "data_processing_configuration": {
4    "id": "3dd973d3fda8604fba040381bfcaab29",
5    "name": "Sample Onboarding Configuration"
6  },
7  "datasource": {
8    "id": "1ea7e575defdf6bc3f26a3f127e98170",
9    "name": "Sample Datasource"
10  },
11  "dataset": {
12    "id": "3c4d09e4ef50b370ae0efacdb43ec2dd",
13    "name": "Sample Dataset"
14  },
15  "id": "7c2e73157a35658c245511bcf2a95195",
16  "name": "New Sample Onboarding Job",
17  "account": {
18    "id": "e268443e43d93dab7ebef303bbe9642f",
19    "name": "Sample Account"
20  },
21  "compute_configuration": {
22    "id": "351df95006b1350e597762a1d58e9319",
23    "name": "Sample Compute Configuration"
24  },
25  "status": "STARTING",
26  "maximum_runtime_seconds": 1,
27  "created_at": "2026-03-15T10:15:30.000000Z",
28  "created_by": {
29    "id": "dad46b2058752ffde5eaf02f1eb6abd5",
30    "name": "Sample User"
31  },
32  "updated_at": "2026-03-15T10:15:30.000000Z",
33  "stage": "sample_stage",
34  "progress": 0.5,
35  "stage_progress": 0.5
36}

Delete a job

Delete an onboarding job when it should no longer be kept.

DELETE
/data-processing-onboarding-jobs/:dataProcessingOnboardingJobId
1curl -X DELETE https://api.hi.umnai.com/data-processing-onboarding-jobs/dataProcessingOnboardingJobId \
2     -H "Authorization: Bearer <token>"