Track 07 · Data & governanceLesson 06 of 11
Lineage: proving where an answer came from
Data quality, lineage and a catalogue that traces any answer to its source.
- Specialisation · Decision platforms
- 30 min read
- 75 min hands-on
- Pro
What you'll be able to do
01- Explain the six quality dimensions (completeness, validity, accuracy, consistency, timeliness, uniqueness) and write a measurable check for each.
- Design a data contract for a dataset an agent depends on, with owners, quality rules and a freshness SLA, using the Open Data Contract Standard as the shape.
- Implement a gate that quarantines bad rows, blocks an agent index refresh when error-level checks fail, and emits OpenLineage run events as evidence.
- Trace an agent answer back to the source rows, snapshot, contract version and pipeline run that produced it.
- Choose where table-level versus column-level lineage is needed, and which catalogue holds it, given what the customer already runs.
- Defend a quality gate to a data-platform owner who fears delays and a sponsor who fears stale answers.
Before you start
- Finding where the data really lives: the source map is the first draft of your lineage graph.
- Is the data ready?: the curated layer the agent reads from.
- Answers with receipts: grounding and citations: the answer contract whose citations this concept resolves to rows.
- Keeping the knowledge base fresh: how indexes are rebuilt; this concept decides whether a rebuild is allowed.
The short version
An agent is a fluent amplifier for whatever data it reads. If you cannot say which rows, from which run, under which contract, produced a given answer, you cannot defend that answer to a model-risk reviewer or fix it when it is wrong. So you write a data contract for every dataset the agent depends on, test it on every load across the six quality dimensions, quarantine the rows that fail, stop the index refresh when the batch fails, and record lineage at table and column level as machine-readable events. The catalogue is where owners, definitions, quality signals and lineage live together so that someone other than you can answer "where did this come from?" next year.
- Quality is measured per dimension, per check, with a threshold and a severity. "The data is good" is not a measurement.
- A contract is an agreement with a named owner, not a schema file. ODCS gives it a portable shape.
- Gate the consumer, not the whole estate. Block the agent index refresh; keep serving the last good snapshot.
- Lineage is evidence only if it is captured at run time and links to the exact snapshot the agent read.
- Every index document carries a source reference (dataset, primary key, row hash, extract time, run ID, contract version). That reference is what makes the trace possible.
Practise this in the lab
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI