Dataset lineage

Trace how a dataset was created from its datasource and raw inputs.
View as Markdown

The Dataset lineage view describes the lineage graph for an onboarded dataset.

Use it to understand which datasource produced a dataset, which raw inputs contributed upstream, and how ingestion and onboarding steps connect the data assets together.

When to use

Use caseDescription
Audit dataset provenanceTrace a dataset back to its datasource and raw inputs.
Review onboarding historyInspect the onboarding relationship between datasource and dataset nodes.
Understand data volumeReview row counts, train/test split counts, masked rows, and size information.
Build lineage visualisationsUse nodes and edges to render a graph of the dataset lineage.

Generate the view

Generate the Dataset lineage view from a dataset.

POST
/datasets/:datasetId/views
1curl -X POST https://api.hi.umnai.com/datasets/datasetId/views \
2 -H "Authorization: Bearer <token>" \
3 -H "Content-Type: application/json" \
4 -d '{
5 "data": [
6 {
7 "view_type": "DATASET_LINEAGE"
8 }
9 ]
10}'
Response
1{
2 "data": [
3 {
4 "view_type": "DATASET_LINEAGE",
5 "view_data": {
6 "nodes": {
7 "columns": [
8 "friendly_label",
9 "node_type",
10 "label",
11 "increment_label",
12 "increment_index",
13 "branch",
14 "source_branch",
15 "n_rows",
16 "n_rows_train",
17 "n_rows_test",
18 "n_rows_masked",
19 "n_input_files",
20 "input_files",
21 "increment_size_mb",
22 "increment_mask_size_mb",
23 "total_increment_size_mb",
24 "increment_run_id",
25 "incremented_at",
26 "datasource_id",
27 "dataset_id",
28 "datasource_name",
29 "dataset_name"
30 ],
31 "index": [
32 0,
33 1,
34 2,
35 3
36 ],
37 "data": [
38 [
39 "2bcb9dc8-adult_income-root",
40 "DATASET",
41 "2bcb9dc8-adult_income-root",
42 "main",
43 0,
44 "main",
45 null,
46 48842,
47 60301,
48 9705,
49 21164,
50 2,
51 "[0 1]",
52 5.7087664604,
53 2.4736974239,
54 8.1824638844,
55 "5786b8a368a6479886efe5482b1882e1",
56 "2026-01-29 17:37:00",
57 "None",
58 "daf6694d509448c9bedd8b624a08999f",
59 "None",
60 "2bcb9dc8-adult_income-root"
61 ],
62 [
63 "6330f1c8-adult_income-root",
64 "DATASOURCE",
65 "6330f1c8-adult_income-root",
66 "main",
67 0,
68 "main",
69 null,
70 48842,
71 39137,
72 9705,
73 0,
74 2,
75 "[0 1]",
76 5.7087664604,
77 0,
78 5.7087664604,
79 "ba4e1314d6a947a2baad05bc0823b2e8",
80 "2026-01-29 17:31:22",
81 "72ee0da340ef4d4b83fb92d23de60db4",
82 "None",
83 "6330f1c8-adult_income-root",
84 "None"
85 ],
86 [
87 "RAW::0",
88 "RAW",
89 "0",
90 "None",
91 null,
92 "None",
93 null,
94 null,
95 null,
96 null,
97 null,
98 null,
99 "None",
100 null,
101 null,
102 null,
103 "None",
104 "NaT",
105 "None",
106 "None",
107 "None",
108 "None"
109 ],
110 [
111 "RAW::1",
112 "RAW",
113 "1",
114 "None",
115 null,
116 "None",
117 null,
118 null,
119 null,
120 null,
121 null,
122 null,
123 "None",
124 null,
125 null,
126 null,
127 "None",
128 "NaT",
129 "None",
130 "None",
131 "None",
132 "None"
133 ]
134 ],
135 "foreign_keys": [],
136 "labels": [],
137 "types": [
138 {
139 "data_type": "STRING",
140 "format": null
141 },
142 {
143 "data_type": "STRING",
144 "format": null
145 },
146 {
147 "data_type": "STRING",
148 "format": null
149 },
150 {
151 "data_type": "STRING",
152 "format": null
153 },
154 {
155 "data_type": "NUMBER",
156 "format": "INT64"
157 },
158 {
159 "data_type": "STRING",
160 "format": null
161 },
162 {
163 "data_type": "ANY"
164 },
165 {
166 "data_type": "NUMBER",
167 "format": "INT64"
168 },
169 {
170 "data_type": "NUMBER",
171 "format": "INT64"
172 },
173 {
174 "data_type": "NUMBER",
175 "format": "INT64"
176 },
177 {
178 "data_type": "NUMBER",
179 "format": "INT64"
180 },
181 {
182 "data_type": "NUMBER",
183 "format": "INT64"
184 },
185 {
186 "data_type": "STRING",
187 "format": null
188 },
189 {
190 "data_type": "NUMBER",
191 "format": "FLOAT64"
192 },
193 {
194 "data_type": "NUMBER",
195 "format": "FLOAT64"
196 },
197 {
198 "data_type": "NUMBER",
199 "format": "FLOAT64"
200 },
201 {
202 "data_type": "STRING",
203 "format": null
204 },
205 {
206 "data_type": "STRING",
207 "format": null
208 },
209 {
210 "data_type": "STRING",
211 "format": null
212 },
213 {
214 "data_type": "STRING",
215 "format": null
216 },
217 {
218 "data_type": "STRING",
219 "format": null
220 },
221 {
222 "data_type": "STRING",
223 "format": null
224 }
225 ]
226 },
227 "edges": {
228 "columns": [
229 "link_from",
230 "link_to",
231 "link_type",
232 "source_branch",
233 "target_branch"
234 ],
235 "index": [
236 0,
237 1,
238 2
239 ],
240 "data": [
241 [
242 "6330f1c8-adult_income-root",
243 "2bcb9dc8-adult_income-root",
244 "ONBOARDING",
245 "main",
246 "main"
247 ],
248 [
249 "RAW::0",
250 "6330f1c8-adult_income-root",
251 "INGESTION",
252 "",
253 "main"
254 ],
255 [
256 "RAW::1",
257 "6330f1c8-adult_income-root",
258 "INGESTION",
259 "",
260 "main"
261 ]
262 ],
263 "foreign_keys": [],
264 "labels": [],
265 "types": [
266 {
267 "data_type": "STRING",
268 "format": null
269 },
270 {
271 "data_type": "STRING",
272 "format": null
273 },
274 {
275 "data_type": "STRING",
276 "format": null
277 },
278 {
279 "data_type": "STRING",
280 "format": null
281 },
282 {
283 "data_type": "STRING",
284 "format": null
285 }
286 ]
287 }
288 },
289 "views_version": {
290 "major_version": 0,
291 "minor_version": 5,
292 "patch_version": 0,
293 "build_version": "dev76"
294 },
295 "output_version": {
296 "major_version": 1,
297 "minor_version": 0,
298 "patch_version": 3,
299 "build_version": null
300 }
301 }
302 ]
303}

The request only needs view_type set to DATASET_LINEAGE.

Output

The response follows the shared Response structure format.

The Dataset lineage view returns multiple named dataframes inside view_data.

DataframeDescription
nodesLineage graph nodes, including dataset, datasource, and raw input nodes.
edgesLineage graph edges connecting raw inputs, datasources, datasets, and lineage relationships.

Interpret the result

The Dataset lineage view is easiest to read as a graph: start with the dataset node, then follow the edges back to the datasource and raw inputs.

Start with the dataset node

Use nodes to find the row where node_type is DATASET.

This row describes the onboarded dataset and includes dataset identifiers, dataset name, row counts, split counts, masked rows, increment metadata, and size information.

Trace the datasource

Use rows where node_type is DATASOURCE to identify the datasource that produced the dataset.

The datasource node helps connect the onboarded dataset back to the ingestion source. This is useful when auditing which datasource version or branch was used to create a dataset.

Identify raw inputs

Use rows where node_type is RAW to identify raw input nodes.

Raw nodes represent upstream input files or raw data sources that contributed to the datasource, and therefore indirectly to the dataset.

Follow edges

Use edges.link_from and edges.link_to to understand the direction of lineage.

For dataset lineage, the graph commonly includes an ONBOARDING edge from datasource to dataset, and INGESTION edges from raw inputs to the datasource. source_branch and target_branch describe the branch relationship when branch information is available.

Review increments and branches

Use increment_label, increment_index, branch, and source_branch to understand how dataset and datasource versions or increments relate to each other.

This is useful when a dataset has been created from a specific branch or when you need to understand which increment produced the dataset currently being used for training or review.

Check data volume

Use n_rows, n_rows_train, n_rows_test, n_rows_masked, n_input_files, increment_size_mb, increment_mask_size_mb, and total_increment_size_mb to review the size and split of each lineage node.

These fields are useful for verifying that the dataset contains the expected amount of data before training, evaluation, or model review.

Field reference

The Dataset lineage view returns dataframes, so individual columns are not documented as standalone API schema properties.

nodes

The nodes dataframe contains the graph nodes.

FieldDescription
friendly_labelHuman-friendly label for the lineage node.
node_typeType of lineage node, such as DATASET, DATASOURCE, or RAW.
labelNode label used to identify the node in the lineage graph.
increment_labelLabel for the dataset or datasource increment, when available.
increment_indexIndex of the increment, when available.
branchBranch associated with the node.
source_branchSource branch associated with the node, when available.
n_rowsNumber of rows associated with the node or increment.
n_rows_trainNumber of training rows associated with the node or increment.
n_rows_testNumber of test rows associated with the node or increment.
n_rows_maskedNumber of masked rows associated with the node or increment.
n_input_filesNumber of input files associated with the node or increment.
input_filesInput file references associated with the node or increment.
increment_size_mbSize of the increment in megabytes.
increment_mask_size_mbSize of the increment mask in megabytes.
total_increment_size_mbTotal size of the increment in megabytes.
increment_run_idID of the run that produced the increment.
incremented_atTimestamp when the increment was created.
datasource_idID of the datasource associated with the node, when available.
dataset_idID of the dataset associated with the node, when available.
datasource_nameName of the datasource associated with the node, when available.
dataset_nameName of the dataset associated with the node, when available.

edges

The edges dataframe contains the graph edges.

FieldDescription
link_fromSource node label for the edge.
link_toTarget node label for the edge.
link_typeType of lineage relationship, such as ONBOARDING or INGESTION.
source_branchSource branch for the edge, when available.
target_branchTarget branch for the edge, when available.