Track 07 · Data & governanceLesson 04 of 11
Streaming data for real-time agents
Events, queues and change capture for agents that react as things happen.
- Lesson library · reference
- 30 min read
- 75 min hands-on
- Pro
What you'll be able to do
01- You can choose batch, micro-batch or streaming from a use case's freshness budget.
- You can explain Kafka's model (partitions, offsets, consumer groups, retention, compaction) to a platform owner.
- You can design a CDC feed, decide when to use an outbox, and evolve schemas without breaking consumers.
- You can implement an idempotent, order-aware consumer with event-time windows, watermarks and a late-data path.
- You can defend filtering and aggregating before any model call on cost and backpressure grounds.
Before you start
- Finding where the data really lives: which system is the source of record for each event.
- Is the data ready?: the batch side the stream is reconciled against.
- Agents started by events and schedules: how an event starts agent work. This file is the data underneath.
- Retries, fallbacks and failing gracefully: idempotency and retries inside the agent.
The short version
Most "real-time agent" requests are really requests for a pipeline that hands the agent a small, recent, trustworthy fact. Events land on a durable, partitioned log (usually Kafka) from applications, database change-data-capture (CDC) or an outbox table. A stream processor deduplicates them, keeps per-key order, windows them by event time and decides when something deserves a model's attention. The agent reads materialised context or receives a trigger, never the raw firehose.
- Start with the freshness budget; many "real-time" asks are met by a five-minute micro-batch.
- Delivery is at-least-once in practice: every consumer and side effect needs an idempotency key.
- Ordering is per partition, so the partition key is your ordering guarantee.
- Watermarks decide when an event-time window is complete; late data needs a defined home.
- A model call per event is a cost bug and a backpressure bug.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI