Track 07 · Data & governanceLesson 06 of 11

Lineage: proving where an answer came from

Data quality, lineage and a catalogue that traces any answer to its source.

What you'll be able to do

What you'll be able to do

01
  • Explain the six quality dimensions (completeness, validity, accuracy, consistency, timeliness, uniqueness) and write a measurable check for each.
  • Design a data contract for a dataset an agent depends on, with owners, quality rules and a freshness SLA, using the Open Data Contract Standard as the shape.
  • Implement a gate that quarantines bad rows, blocks an agent index refresh when error-level checks fail, and emits OpenLineage run events as evidence.
  • Trace an agent answer back to the source rows, snapshot, contract version and pipeline run that produced it.
  • Choose where table-level versus column-level lineage is needed, and which catalogue holds it, given what the customer already runs.
  • Defend a quality gate to a data-platform owner who fears delays and a sponsor who fears stale answers.
Before you start

Before you start

The short version

The short version

An agent is a fluent amplifier for whatever data it reads. If you cannot say which rows, from which run, under which contract, produced a given answer, you cannot defend that answer to a model-risk reviewer or fix it when it is wrong. So you write a data contract for every dataset the agent depends on, test it on every load across the six quality dimensions, quarantine the rows that fail, stop the index refresh when the batch fails, and record lineage at table and column level as machine-readable events. The catalogue is where owners, definitions, quality signals and lineage live together so that someone other than you can answer "where did this come from?" next year.

  • Quality is measured per dimension, per check, with a threshold and a severity. "The data is good" is not a measurement.
  • A contract is an agreement with a named owner, not a schema file. ODCS gives it a portable shape.
  • Gate the consumer, not the whole estate. Block the agent index refresh; keep serving the last good snapshot.
  • Lineage is evidence only if it is captured at run time and links to the exact snapshot the agent read.
  • Every index document carries a source reference (dataset, primary key, row hash, extract time, run ID, contract version). That reference is what makes the trace possible.

Practise this in the lab

Pro lesson

The rest of this lesson is in Pro.

The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.

  • Every stage and specialisation, in full
  • Every lab and every role project
  • Halden weeks 2–10, on your record
  • Every drill, live in the playground
  • All solutions, plus the server judge
  • Lab Assistant and every Teach with AI