Track 03 · Agents & toolsLesson 11 of 12

Rules, SQL, models, solvers or an LLM?

Pick the cheapest technique that solves each decision, and use the LLM where only it can.

The FDE path · Stage 1 of 6
35 min read
75 min hands-on

What you'll be able to do

Break a request such as "an AI agent for cut-off night" into decisions, each with how often it is made, what a wrong answer costs and what data exists for it.
Choose a technique per decision from the ladder (rule, query, statistical baseline, trained model, solver, LLM), and say why the cheaper rungs don't fit before you climb.
Build the baseline first and make every cleverer method beat it on data it hasn't seen.
Evaluate each technique with its own metric: backtested MAE, precision and recall at a fixed workload, calibration, plan cost and constraint violations.
Combine techniques so the LLM reads text into typed constraints and drafts explanations, while validated code, models and solvers decide.
Defend the design on cost, accuracy, explainability and change to a sponsor who asked for "AI".

Before you start

Shadowing the work as it's really done and Finding the real requirement: the decision inventory is built from what you saw and were told.
The operational model: objects, links and actions: the exceptions, operators and actions the plan refers to.
Baselines and ROI you can defend: how the desk performs today. This lesson adds the technical baseline, the simplest method a cleverer one must beat.
Comfortable Python and school algebra; no machine-learning or operations-research background.

Starting here? On the core path this lesson closes stage 1, before weeks 1 and 3. Two ideas used below: a structured output is a model response forced into a JSON Schema so code can validate it (Typed output: schemas, validation and tool calls); a workflow is code that decides the order of steps, with a model filling in some of them (Agent or workflow? Choosing the autonomy level).

The short version

A request for "an agent" usually bundles several decisions, each with a cheapest technique that solves it well. List the decisions first, then climb a ladder only as far as each one needs: a written rule, a query, a statistical baseline, a trained model on tabular data, a solver for plans under constraints, and a language model for reading and writing language. The LLM is often essential, but in a narrow role: it turns messy text into typed inputs that code validates, and turns a solver's output into words a person can check. It doesn't do arithmetic over hundreds of records, enforce hard constraints or decide who works what.

One technique per decision. "Use AI" isn't a design; "seasonal naive, logistic regression, an assignment solver and an LLM extractor" is.
Baselines first. A method that can't beat seasonal naive or the desk's own rule isn't ready.
Each part gets its own metric, measured against its own baseline.
Hard constraints live in code. A solver respects them by construction and a checker verifies them; a prompt only asks.
The LLM reads and writes; deterministic parts decide.

Mental model

Picture a port control room the night before a big sailing. There is a tide table on the wall (a forecast), a duty officer with a checklist ("anything hazardous goes to the senior pilot"), a berth planner with hard rules (two ships can't share a berth), and a radio operator who listens to the ships, writes down what they need in a fixed format and reads the plan back. The radio operator is essential, because ships speak in many accents and half-sentences. But nobody asks the radio operator to work out the berth plan, and nothing heard on the radio changes the plan until it has been written down and checked.

A system for Halden's cut-off night has the same shape.

Diagram

flowchart LR
  H["History in Snowflake<br/>exceptions per day, outcomes"] --> F["Forecast<br/>seasonal naive"]
  H --> S["Score<br/>logistic regression"]
  M["Emails to the desk"] --> X["LLM extracts<br/>typed constraints"]
  X --> V{"Validate<br/>in code"}
  V -- rejected --> P["A person decides"]
  V -- accepted --> O["Solver<br/>assign operators to exceptions"]
  S --> O
  R["Roster: skills,<br/>languages, shifts"] --> O
  F --> D["Staffing check<br/>for the night"]
  O --> C["Checker recomputes<br/>cost and constraints"]
  C --> E["Explanation<br/>template or LLM rephrase"]
  E --> L["Desk lead approves<br/>the plan"]

Only two boxes contain a language model, and neither decides anything on its own.

Core concepts

The decision inventory

Start from what you saw on the desk and turn the request into a list of decisions, each with four fields.

Field	What to write	Cut-off night example
The decision	A question with a typed answer	"Which operator works EXC-10460, and when?" (an operator and a time slot)
How often	Per day, per item, per event	About 30 assignments every Thursday night
Cost of being wrong	In money, time, risk or trust, and who bears it	A rolled container: rebooking, storage and a penalty (US$1,800–4,500 in the example's invented figures)
Data and inputs that exist	History, labels, rules, constraints, free text	11 weeks of exceptions with outcomes; the roster; emails

Then add who decides today and who must be able to explain it: a decision someone can challenge needs a technique whose reasoning you can show. Cut-off night has five decisions: how busy the night will be, which exceptions matter most, what the emails change, who works what, and how to explain the plan. Only the emails are a language problem.

The technique ladder

Climb only as far as the decision needs. Each rung costs more to build, run and explain than the one below.

Rung	Good at	Data it needs	How it fails
Rule (if-then, lookup)	policy already written down; thresholds	none beyond the rule	brittle at the edges; rules pile up and contradict
Query (SQL, an aggregate)	counting, filtering, joining: "how many, which"	the tables	wrong definitions; stale data
Statistical baseline (seasonal naive, base rate)	forecasts and scores with little data; the bar to beat	a short history	misses effects it doesn't model
Trained model (logistic regression, boosted trees)	scoring rows with many weak signals	hundreds to thousands of labelled rows	drift; leakage; badly calibrated probabilities
Solver (assignment, MIP, constraint programming)	the best plan under hard constraints	the objective and every constraint	a wrong or missing constraint gives a confident wrong plan
LLM	text in, language out; extraction; ambiguity	instructions, examples and an eval set	plausible errors; arithmetic slips; not repeatable

Cost and explainability rise with the ladder. Every rung below the LLM costs close to nothing per run, and you explain it by reading it (a rule, a query) or by showing its coefficients or the constraints that bind (a model, a solver). An LLM costs the most per call, adds latency, and is explained only by quoting its input and output.

Two rungs are easy to skip. Queries answer a surprising share of "AI" requests: "exceptions older than two days from a shipper on the list" is a WHERE clause. Solvers are the rung most software engineers have never used, so plans get built by hand-written greedy loops or by asking an LLM to "allocate sensibly". If the answer is a plan, schedule, route or allocation that must respect rules, it is a solver's job.

Baselines first

A baseline is the simplest method that gives a real answer. It is the bar a cleverer method must clear on held-out data, the fallback when that method breaks, and often good enough to ship.

Forecasts: naive (tomorrow equals today), seasonal naive (this Thursday equals last Thursday), a recent mean.
Classification: the majority class, the base rate as a probability, and the rule the desk already uses. The last matters politically: your model replaces Rafaela's list, so it has to beat Rafaela's list.
Plans: the desk's habit ("riskiest first, to whoever is free"), written down as a greedy loop.

Put the baseline's numbers in the same table as the candidate's, on the same test data. If the gain doesn't matter at the decision's cost, ship the baseline.

Forecasting basics

A forecast answers "how many, when" for a future period.

Horizon and granularity. Forecast what the decision needs: exceptions per day for the next week to staff the desk, not per hour for a month.
Seasonality. Halden's exceptions pile up before each cut-off and nearly stop at weekends. Seasonal naive captures that cycle for free, which is why it is hard to beat on operational data.
Backtesting. Never judge a forecast by its fit to the past. Use rolling origin evaluation: from a past Sunday, forecast the next week with data up to then, compare with what happened, move the origin a week, repeat.
Error measures. MAE (mean absolute error) is in the units people plan with. MAPE (mean absolute percentage error) is popular and misleading here: undefined when the actual is zero, explosive on small counts, and harder on over-forecasts than under-forecasts. Hyndman and Athanasopoulos's textbook (Further reading) sets out these problems and uses scaled errors such as MASE, which divide by a naive method's training error, to compare series of different sizes.

Next come exponential smoothing with seasonality (Holt-Winters, ETS), regression on calendar features and boosted trees over lagged values, each earning its place in the same backtest.

Classification on tables

Most operational scoring is a classifier over rows with a few dozen columns: will this exception make the container miss its sailing?

Logistic regression comes first: fast, stable on small data, and its coefficients become odds ratios a desk lead can read ("a customs exception multiplies the odds of a roll by 3.5").
Gradient-boosted trees (XGBoost, LightGBM, scikit-learn's HistGradientBoostingClassifier) are the usual strong model; a 2022 benchmark across 45 datasets (Further reading) found tree-based models still ahead of deep learning on medium-sized tabular data. Explain them with feature-effect plots and worked cases.
Evaluate at the workload you can staff. The desk can work 30 exceptions tonight, so ask: of those flagged, how many would really have rolled (precision), and of all that would have rolled, how many were flagged (recall)? Accuracy is meaningless when two thirds of exceptions resolve on their own.
Calibration. If the model says 0.8, about 80% of such cases should roll. It matters whenever a probability is multiplied by money. The Brier score (mean squared error of the probabilities) is a quick check against always predicting the base rate.
Leakage. A feature known only after the outcome ("rebooking requested") makes a model brilliant in testing and useless live. Ask of every column: would I have known this at 22:00 on the night? And split train and test by time, not at random.

Optimisation

A solver chooses values for decision variables to minimise an objective subject to constraints. Assigning operators to exceptions is the classic case:

Variables: for each exception and each operator-hour slot, yes or no.
Objective: the expected roll cost left after the night: each exception's risk times the share that remains if it is worked in that slot (less slack before its deadline leaves more), or its full risk if nobody works it.
Constraints: one exception per slot, one slot per exception; the operator has the skill, speaks the language and is on shift; the slot ends before the exception is due.

With only "each row once, each column once" plus forbidden pairs, this is a pure assignment problem, which the Hungarian algorithm solves exactly in polynomial time: about 35 lines in the worked example, scipy.optimize.linear_sum_assignment or OR-Tools' linear sum assignment solver in production.

Real plans grow side constraints that span many assignments: "the customs broker takes at most five submissions tonight". Those break the pure assignment structure, and the tools become a mixed-integer programming (MIP) solver such as HiGHS or a constraint programming solver such as OR-Tools CP-SAT. OR-Tools' documentation calls its linear sum assignment solver faster for the simple problem and its MIP and CP-SAT solvers far more general, with an "assignment with teams" example that adds this kind of shared limit (as of October 2026; OR-Tools is Apache-2.0 licensed, HiGHS MIT).

Two more ideas for site. Feasibility first: when no plan meets every constraint, report which ones conflict so a person can relax one ("Márcio stays an hour late"); a "not covered" option with a cost lets the plan degrade instead of failing. Heuristics when exact is too slow: greedy plus local search is fine at very large scale, once you have measured its gap to an exact answer on smaller cases.

Where LLMs are strong, and where they are weak

Strong: reading emails, documents and chat in many languages; extracting typed fields; classifying fuzzy free text; drafting messages; explaining a result in plain words; odd cases nobody wrote a rule for.
Weak: arithmetic over many records; hard constraints across a plan; exact repeatability, which audit and trust need; optimisation, where an LLM's "allocation" is a plausible guess, not a minimum.

The test: if you can write the procedure down and run it in code, run it in code. Use the model where nobody can write the procedure down, which is usually where people's language comes in.

Hybrid patterns

Useful designs combine rungs, with a validated hand-off between them.

LLM extracts, code validates, solver plans, template explains. The worked example's pattern. The extractor returns typed objects through a JSON Schema; a validator rejects anything that doesn't fit tonight's facts; the explanation quotes the solver's inputs.
Model scores, rules threshold, a person decides. The default for consequential decisions (Approval gates and bounded autonomy).
LLM proposes, solver checks. For open-ended changes ("rearrange tomorrow's shifts"), the model suggests a typed change; the solver decides whether it is feasible and what it costs.
Forecast feeds the plan. Tonight's expected volume sets how many operators to call in, which sets the solver's capacity.

In each, the hand-off is a data contract, not a prompt: the determinism boundary of Agent or workflow? Choosing the autonomy level, drawn one step earlier, to decide whether a model call belongs in the step at all.

Evaluating each part

Part	Metric	Baseline	Test data
Forecast	MAE (MASE across lanes), rolling-origin backtest	seasonal naive	the last four to eight weeks
Score	precision and recall at the staffed workload; Brier score	the desk's rule; the base rate	a later period than training
Extractor	per-field accuracy; rejection rate; zero invented fields	a regex, or nothing	labelled emails (Golden datasets and the metrics that matter)
Plan	objective value; constraint violations (must be zero)	the desk's greedy habit	replayed past nights
End to end	containers rolled per cut-off; desk hours; overrides	last quarter	a shadow run beside the desk

Cost, explainability and change

Cost. Everything below the LLM rung runs in milliseconds on a laptop. An LLM call per email is affordable; an LLM inside a planning loop is not, and would still be wrong.
Explainability. "She got it because it needs customs skill, it's due at 12:00, and the earlier customs slots went to riskier holds" satisfies an auditor. "The model thought it was best" doesn't.
Change. When Halden adds a broker limit, a language or a night shift, the solver takes one more constraint and the checker one more assertion. A prompt that "allocates sensibly" has to be re-tested from scratch.

Worked example

Halden's cut-off night: one problem, five techniques. It is Wednesday 25 March 2026, 22:00 in São Paulo; Santos closes for the Rotterdam sailing at 15:00 on Thursday. The program forecasts the week's exceptions, scores tonight's open exceptions for the chance their container misses the sailing, reads four emails into typed constraints, assigns six operators to the 30 riskiest exceptions (greedy, then exact), and explains the cases a desk lead will ask about.

Requirements: Python 3.11+, standard library only. Run python3 cutoff_night.py; it takes about a second. All data is generated from a fixed seed and invented; the shippers, operators, languages, skills and shifts match the Halden estate (_lab/estates/halden), so the exercise can move onto the full estate.

python

"""Cut-off night at Halden's São Paulo desk: one problem, five techniques.

Forecast the load, score the risk, read the emails, plan the desk, explain the plan.
Standard library only, Python 3.11+. Every name, number and email is invented; the
shippers, operators and shifts match the Halden estate in _lab/estates/halden.
"""
from __future__ import annotations

import math
import random
import re
from dataclasses import dataclass
from datetime import datetime, timedelta

RNG = random.Random(15)
HISTORY_DAYS = 77                    # Monday 5 January (the estate's first day) to Sunday 22 March
NIGHT = datetime(2026, 3, 25, 22)    # Wednesday 22:00 in São Paulo (BRT, no summer time)
CUTOFF = datetime(2026, 3, 26, 15)   # Thursday 15:00: the Santos cut-off for Rotterdam


def poisson(lam: float) -> int:
    """Knuth's method; fine for the small rates here."""
    limit, k, p = math.exp(-lam), 0, 1.0
    while True:
        p *= RNG.random()
        if p <= limit:
            return k
        k += 1


# --- 1. Forecast: how many exceptions does each lane open per day? ---------------

# Mean exceptions opened per weekday, Monday first: they pile up before each cut-off
# (Santos closes Thursday, Jakarta early Wednesday local time) and almost stop at weekends.
WEEKLY = {"Santos → Rotterdam": (9, 11, 16, 22, 12, 4, 2),
          "Jakarta → Jebel Ali": (8, 15, 9, 6, 5, 2, 1)}
COUNTS = {lane: [poisson(m[d % 7] * (1 + 0.004 * d)) for d in range(HISTORY_DAYS)]
          for lane, m in WEEKLY.items()}


def seasonal_naive(train: list[int], h: int) -> list[float]:
    return [float(train[len(train) - 7 + (i % 7)]) for i in range(h)]


def ses(train: list[int], h: int) -> list[float]:
    """Simple exponential smoothing, alpha chosen by one-step error on the training data."""
    def run(alpha: float) -> tuple[float, float]:
        level, sse = float(train[0]), 0.0
        for y in train[1:]:
            sse += (y - level) ** 2
            level = alpha * y + (1 - alpha) * level
        return sse, level
    best = min((a / 20 for a in range(1, 20)), key=lambda a: run(a)[0])
    return [run(best)[1]] * h


def backtest(series: list[int], method) -> tuple[float, float, int]:
    """Rolling origin over the last four weeks, a week ahead each time: MAE, MAPE, zero days."""
    errors, pct, zeros = [], [], 0
    for origin in range(HISTORY_DAYS - 28, HISTORY_DAYS, 7):
        for f, y in zip(method(series[:origin], 7), series[origin:origin + 7]):
            errors.append(abs(y - f))
            if y == 0:
                zeros += 1          # MAPE is undefined here, so it silently skips the day
            else:
                pct.append(abs(y - f) / y)
    return math.fsum(errors) / len(errors), 100 * math.fsum(pct) / len(pct), zeros


# --- 2. Score: which open exceptions will make a box miss the sailing? ------------

FEATURES = ("known_bad", "customs", "docs_missing", "reefer", "days_open", "replied")
_TRUE = (-2.2, (1.1, 1.3, 0.6, 0.4, 0.25, -1.0))  # how the invented history was generated


def features() -> tuple[float, ...]:
    customs = RNG.random() < 0.2
    return (float(RNG.random() < 0.25), float(customs), float(RNG.choice((0, 0, 0, 1, 1, 2))),
            float(RNG.random() < 0.3), float(RNG.randint(0, 6)), float(RNG.random() < 0.4))


def sigmoid(z: float) -> float:
    return 1 / (1 + math.exp(-z))


HISTORY = []  # (features, 1 if the box missed its sailing), oldest first
for _ in range(900):
    x = features()
    HISTORY.append((x, int(RNG.random() < sigmoid(_TRUE[0] + math.fsum(w * v for w, v in zip(_TRUE[1], x))))))
TRAIN, TEST = HISTORY[:630], HISTORY[630:]


def rule(x: tuple[float, ...]) -> bool:
    return x[0] == 1.0  # the desk's rule: anything from a shipper on Rafaela's list


def fit(rows, steps: int = 600, lr: float = 0.5, l2: float = 1e-3):
    """Logistic regression by batch gradient descent on standardised features."""
    k = len(rows[0][0])
    mean = [math.fsum(x[j] for x, _ in rows) / len(rows) for j in range(k)]
    std = [math.sqrt(math.fsum((x[j] - mean[j]) ** 2 for x, _ in rows) / len(rows)) or 1.0 for j in range(k)]
    data = [([(x[j] - mean[j]) / std[j] for j in range(k)], y) for x, y in rows]
    w, b = [0.0] * k, 0.0
    for _ in range(steps):
        gw, gb = [l2 * wj for wj in w], 0.0
        for z, y in data:
            err = sigmoid(b + math.fsum(wj * zj for wj, zj in zip(w, z))) - y
            gb += err / len(data)
            for j in range(k):
                gw[j] += err * z[j] / len(data)
        w, b = [wj - lr * g for wj, g in zip(w, gw)], b - lr * gb
    return lambda x: sigmoid(b + math.fsum(wj * (x[j] - mean[j]) / std[j] for j, wj in enumerate(w))), \
        {name: math.exp(wj / std[j]) for j, (name, wj) in enumerate(zip(FEATURES, w))}


def precision_recall(flags: list[bool], ys: list[int]) -> tuple[float, float]:
    tp = sum(f and y for f, y in zip(flags, ys))
    return tp / max(1, sum(flags)), tp / max(1, sum(ys))


# --- 3. Tonight's open exceptions ---------------------------------------------------

SHIPPERS = (("Cafés Serra Alta", False), ("Citrus Paulista Exportação", True), ("Vinícola Vale do Sol", True),
            ("Pneus Atlântico", False), ("Confecções Aurora", False), ("Têxtil Santa Clara", False),
            ("Café Mogiana Trading", False), ("Frutas do São Francisco", True),
            ("Malharia Ipê Amarelo", False), ("Rodas & Pneus Litoral", False))  # (name, ships reefer)
KNOWN_BAD = {"Citrus Paulista Exportação", "Confecções Aurora", "Malharia Ipê Amarelo"}
SKILL = {"INVOICE_MISMATCH": "invoice", "MISSING_DOCS": "documents",
         "CUSTOMS_HOLD": "customs", "WRONG_HS_CODE": "customs"}  # ROLLED_BOOKING: anyone
DUE = {"CUSTOMS_HOLD": 12, "WRONG_HS_CODE": 12, "MISSING_DOCS": 13, "INVOICE_MISMATCH": 14, "ROLLED_BOOKING": 15}
MISS_COST = {False: 1800, True: 4500}  # US$ if the box rolls: rebooking, storage, penalty (invented)


@dataclass
class Exc:
    id: str
    shipper: str
    type: str
    lang: str
    x: tuple[float, ...]
    due: datetime
    p: float = 0.0
    risk: float = 0.0


def tonight(n: int = 70) -> list[Exc]:
    out = []
    for i in range(n):
        shipper, reefer = RNG.choice(SHIPPERS)
        kind = RNG.choices(tuple(DUE), (0.12, 0.10, 0.20, 0.50, 0.08))[0]
        x = (float(shipper in KNOWN_BAD), float(SKILL.get(kind) == "customs"),
             float(RNG.randint(1, 2) if kind == "MISSING_DOCS" else RNG.choice((0, 0, 0, 1))),
             float(reefer), float(RNG.randint(0, 6)), float(RNG.random() < 0.4))
        due = CUTOFF.replace(hour=DUE[kind]) - timedelta(hours=RNG.choice((0, 0, 1)))
        out.append(Exc(f"EXC-{10401 + i}", shipper, kind, RNG.choices(("pt", "en"), (0.7, 0.3))[0], x, due))
    return out


# --- 4. Emails become typed constraints --------------------------------------------

EMAILS = (
    ("rafaela.souza@halden.example", "pt", "Pessoal, amanhã só chego às 10h (consulta médica). Rafaela"),
    ("financeiro@citruspaulista.example", "pt",
     "Prezados, precisamos da fatura corrigida do EXC-10460 até 26/03 às 11h, senão perdemos a reserva."),
    ("ops@brinkvos.example", "en",
     "Re EXC-10422: our Rotterdam team will take the call, so please have someone call us in English."),
    ("export@vinicolavaledosol.example", "pt",
     "Bom dia, o EXC-10423 pode esperar: precisamos da resposta até 27/03 às 9h. Obrigado."),
)
DIRECTORY = {"rafaela.souza@halden.example": "OP-SP-01"}


@dataclass(frozen=True)
class AvailableFrom:
    operator: str
    at: datetime


@dataclass(frozen=True)
class DueBy:
    exc: str
    at: datetime


@dataclass(frozen=True)
class SpeaksTo:
    exc: str
    lang: str


def extract_constraints(sender: str, lang: str, body: str) -> list:
    """Stand-in for a model call with a typed output schema (W01-06). It returns what a
    good extraction returns, so the example runs offline; the validation below is real."""
    out: list = []
    if m := re.search(r"só chego às (\d{1,2})h", body):
        out.append(AvailableFrom(DIRECTORY.get(sender, "?"), CUTOFF.replace(hour=int(m.group(1)))))
    if m := re.search(r"(EXC-\d{5}).*?até (\d{2})/(\d{2}) às (\d{1,2})h", body):
        out.append(DueBy(m.group(1), datetime(2026, int(m.group(3)), int(m.group(2)), int(m.group(4)))))
    if m := re.search(r"(EXC-\d{5}).*in (English|Portuguese)", body):
        out.append(SpeaksTo(m.group(1), {"English": "en", "Portuguese": "pt"}[m.group(2)]))
    return out


def validate(c, plan_ids: set[str], operators: dict) -> str | None:
    """Deterministic checks on whatever the extractor returned. None means accepted."""
    if isinstance(c, AvailableFrom) and c.operator not in operators:
        return "unknown operator"
    if isinstance(c, (DueBy, SpeaksTo)) and c.exc not in plan_ids:
        return f"{c.exc} is not in tonight's plan"
    if isinstance(c, (AvailableFrom, DueBy)) and not NIGHT <= c.at <= CUTOFF:
        return "time outside tonight's window"
    if isinstance(c, SpeaksTo) and c.lang not in ("pt", "en", "es"):
        return "unsupported language"
    return None


# --- 5. Plan: who works which exception, and when? ----------------------------------

OPERATORS = {  # from the estate's operators.csv: name, languages, skills, shift start, length
    "OP-SP-01": ("Rafaela Souza", {"pt", "en", "es"}, {"invoice", "customs", "reefer"}, 8, 9),
    "OP-SP-02": ("João Ferreira", {"pt", "en"}, {"invoice", "documents"}, 6, 9),
    "OP-SP-03": ("Camila Rocha", {"pt", "es", "en"}, {"customs", "reefer"}, 8, 9),
    "OP-SP-04": ("Thiago Albuquerque", {"pt"}, {"invoice"}, 12, 9),
    "OP-SP-05": ("Luíza Gonçalves", {"pt", "en"}, {"documents", "customs"}, 14, 9),
    "OP-SP-06": ("Márcio Tavares", {"pt", "en"}, {"reefer", "invoice"}, 22, 9),
}
# Share of the risk left after an operator works the exception, by hours of slack before it is
# due (invented, but the shape is the desk's experience: last-minute fixes often fail).
RESIDUAL = {0: 0.45, 1: 0.25, 2: 0.15}  # 2 means two hours or more


def slots(available: dict[str, datetime]) -> list[tuple[str, datetime]]:
    """One-hour slots for each operator inside tonight's window and their shift."""
    out = set()
    for op, (_, _, _, start, length) in OPERATORS.items():
        for day in (NIGHT, CUTOFF):  # Wednesday's shift and Thursday's
            t = day.replace(hour=start)
            for h in range(length):
                s = t + timedelta(hours=h)
                if NIGHT <= s and s + timedelta(hours=1) <= CUTOFF and s >= available.get(op, NIGHT):
                    out.add((op, s))
    return sorted(out, key=lambda o: (o[1], o[0]))


def feasible(e: Exc, slot: tuple[str, datetime]) -> bool:
    _, langs, skills, _, _ = OPERATORS[slot[0]]
    return (SKILL.get(e.type) in skills | {None} and e.lang in langs
            and slot[1] + timedelta(hours=1) <= e.due)


def cost(e: Exc, slot: tuple[str, datetime]) -> float:
    """Expected roll cost left (US$) if this slot works this exception."""
    slack = int((e.due - slot[1] - timedelta(hours=1)).total_seconds() // 3600)
    return e.risk * RESIDUAL[min(slack, 2)]


def greedy(excs: list[Exc], free: list[tuple[str, datetime]]) -> dict[str, tuple[str, datetime] | None]:
    """The desk's habit: riskiest first, to whoever qualified is free soonest."""
    taken, plan = set(), {}
    for e in sorted(excs, key=lambda e: -e.risk):
        options = [s for s in free if s not in taken and feasible(e, s)]
        plan[e.id] = options[0] if options else None
        if options:
            taken.add(options[0])
    return plan


def hungarian(c: list[list[float]]) -> list[int]:
    """Minimum-cost assignment of every row to a distinct column (rows <= columns), O(n^2 m)."""
    n, m, inf = len(c), len(c[0]), float("inf")
    u, v, p, way = [0.0] * (n + 1), [0.0] * (m + 1), [0] * (m + 1), [0] * (m + 1)
    for i in range(1, n + 1):
        p[0], j0, minv, used = i, 0, [inf] * (m + 1), [False] * (m + 1)
        while p[j0]:
            used[j0], i0, delta, j1 = True, p[j0], inf, 0
            for j in range(1, m + 1):
                if not used[j]:
                    cur = c[i0 - 1][j - 1] - u[i0] - v[j]
                    if cur < minv[j]:
                        minv[j], way[j] = cur, j0
                    if minv[j] < delta:
                        delta, j1 = minv[j], j
            for j in range(m + 1):
                if used[j]:
                    u[p[j]] += delta
                    v[j] -= delta
                else:
                    minv[j] -= delta
            j0 = j1
        while j0:
            j1 = way[j0]
            p[j0], j0 = p[j1], j1
    col = [0] * n
    for j in range(1, m + 1):
        if p[j]:
            col[p[j] - 1] = j - 1
    return col


def optimal(excs: list[Exc], free: list[tuple[str, datetime]]) -> dict[str, tuple[str, datetime] | None]:
    big = 1e12  # forbidden pairs; a "miss" column is always allowed, so none is ever chosen
    rows = [[cost(e, s) if feasible(e, s) else big for s in free] + [e.risk] * len(excs) for e in excs]
    col = hungarian(rows)
    return {e.id: free[j] if j < len(free) else None for e, j in zip(excs, col)}


def check(plan, excs: list[Exc]) -> tuple[float, list[str]]:
    """Recompute the plan's cost and verify every constraint, independently of the solver."""
    used = [s for s in plan.values() if s]
    assert len(used) == len(set(used)), "an operator is double-booked"
    total, missed = 0.0, []
    for e in excs:
        s = plan[e.id]
        if s is None:
            total, missed = total + e.risk, missed + [e.id]
        else:
            assert feasible(e, s), f"{e.id} breaks a constraint"
            total += cost(e, s)
    return total, missed


def explain(e: Exc, s: tuple[str, datetime] | None, free: list, notes: dict[str, str]) -> str:
    """The template a model may rephrase; every fact in it comes from the solver's inputs."""
    head = f"{e.id} {e.type.lower().replace('_', ' ')} ({e.shipper}, {e.lang}, due {e.due:%H:%M})"
    why = f"risk US${e.risk:,.0f} = p {e.p:.2f} × US${MISS_COST[bool(e.x[3])]:,}"
    extra = f"; {notes[e.id]}" if e.id in notes else ""
    if s is None:
        n = sum(feasible(e, f) for f in free)
        return f"{head}: NOT COVERED, its {n} qualified slots save more on other work{extra}; {why}"
    return f"{head} → {OPERATORS[s[0]][0]}, {s[1]:%a %H:%M}{extra}; {why}"


if __name__ == "__main__":
    print("1. FORECAST  exceptions opened per day, backtested a week ahead over the last 4 weeks")
    for lane, series in COUNTS.items():
        for name, method in (("seasonal naive", seasonal_naive), ("exp. smoothing", ses)):
            mae, mape, zeros = backtest(series, method)
            print(f"   {lane:20} {name:15} MAE {mae:5.2f}   MAPE {mape:5.1f}%  ({zeros} zero days skipped)")
    wed, thu = (seasonal_naive(COUNTS["Santos → Rotterdam"], 7)[d] for d in (2, 3))
    print(f"   Santos, Wed + Thu this week (seasonal naive): {wed + thu:.0f} new exceptions expected")

    print("\n2. SCORE  will this exception make the box miss the sailing? (270 held-out exceptions)")
    ys = [y for _, y in TEST]
    flags = [rule(x) for x, _ in TEST]
    model, odds = fit(TRAIN)
    probs = [model(x) for x, _ in TEST]
    cut = sorted(probs, reverse=True)[sum(flags) - 1]  # flag as many as the rule does
    for name, f in (("rule: shipper on the list", flags), ("logistic regression", [q >= cut for q in probs])):
        pr, rc = precision_recall(f, ys)
        print(f"   {name:26} flags {sum(f):3}   precision {pr:.2f}   recall {rc:.2f}")
    brier = math.fsum((q - y) ** 2 for q, y in zip(probs, ys)) / len(ys)
    base = math.fsum(y for _, y in TRAIN) / len(TRAIN)
    print(f"   Brier score {brier:.3f} (always guessing the base rate {base:.2f}: "
          f"{math.fsum((base - y) ** 2 for y in ys) / len(ys):.3f})")
    print("   odds ratios: " + ", ".join(f"{k} ×{v:.2f}" for k, v in odds.items()))

    excs = tonight()
    for e in excs:
        e.p = model(e.x)
        e.risk = e.p * MISS_COST[bool(e.x[3])]
    plan_excs = sorted(excs, key=lambda e: -e.risk)[:30]
    ids = {e.id for e in plan_excs}
    print(f"   tonight: {len(excs)} open; the 30 riskiest hold US${math.fsum(e.risk for e in plan_excs):,.0f} "
          f"of expected roll cost and get an operator-hour each")

    print("\n3. READ  emails to the desk become typed constraints")
    available, notes = {}, {}
    for sender, lang, body in EMAILS:
        for c in extract_constraints(sender, lang, body):
            problem = validate(c, ids, OPERATORS)
            print(f"   {'REJECTED ' + problem if problem else 'accepted'}: {c}")
            if problem:
                continue
            if isinstance(c, AvailableFrom):
                available[c.operator] = c.at
            target = next((e for e in plan_excs if e.id == getattr(c, "exc", None)), None)
            if isinstance(c, DueBy) and target:
                target.due = min(target.due, c.at)
                notes[c.exc] = f"deadline from {sender.split('@')[1]}'s email"
            if isinstance(c, SpeaksTo) and target:
                target.lang = c.lang
                notes[c.exc] = f"{c.lang} at {sender.split('@')[1]}'s request"

    print("\n4. PLAN  30 exceptions, 6 operators, one-hour slots from Wed 22:00 to the 15:00 cut-off")
    free = slots(available)
    results = {}
    for name, solve in (("greedy (riskiest first)", greedy), ("optimal (Hungarian)", optimal)):
        plan = solve(plan_excs, free)
        total, missed = check(plan, plan_excs)
        results[name] = (plan, total, missed)
        print(f"   {name:24} {len(free)} slots   cost {total:9,.0f}   missed {len(missed)}: "
              f"US${math.fsum(e.risk for e in plan_excs if e.id in missed):,.0f} at risk")

    print("\n5. EXPLAIN  the optimal plan, for the cases a person will ask about")
    g_plan, g_total, g_missed = results["greedy (riskiest first)"]
    o_plan, o_total, o_missed = results["optimal (Hungarian)"]
    for e in (e for e in plan_excs if e.id in notes or e.id in g_missed):
        print("   " + explain(e, o_plan[e.id], free, notes))
        if e.id in g_missed and e.id not in o_missed:
            print("     (greedy left it uncovered)")

    assert o_total <= g_total + 1e-6 and len(o_missed) <= len(g_missed)
    rejected = [c for s, l, b in EMAILS for c in extract_constraints(s, l, b) if validate(c, ids, OPERATORS)]
    assert len(rejected) == 1 and isinstance(rejected[0], DueBy), "only the Friday deadline goes to a person"
    print("\nOK: optimal is never worse than greedy, and every constraint was checked outside the solver")

What the output shows.

text

1. FORECAST  exceptions opened per day, backtested a week ahead over the last 4 weeks
   Santos → Rotterdam   seasonal naive  MAE  3.93   MAPE  31.4%  (0 zero days skipped)
   Santos → Rotterdam   exp. smoothing  MAE 12.00   MAPE  73.2%  (0 zero days skipped)
   Jakarta → Jebel Ali  seasonal naive  MAE  2.50   MAPE  34.3%  (1 zero days skipped)
   Jakarta → Jebel Ali  exp. smoothing  MAE  4.71   MAPE  67.8%  (1 zero days skipped)
   Santos, Wed + Thu this week (seasonal naive): 55 new exceptions expected

2. SCORE  will this exception make the box miss the sailing? (270 held-out exceptions)
   rule: shipper on the list  flags  75   precision 0.61   recall 0.46
   logistic regression        flags  76   precision 0.68   recall 0.51
   Brier score 0.183 (always guessing the base rate 0.33: 0.236)
   odds ratios: known_bad ×3.10, customs ×3.47, docs_missing ×2.12, reefer ×1.24, days_open ×1.22, replied ×0.34
   tonight: 70 open; the 30 riskiest hold US$54,008 of expected roll cost and get an operator-hour each

3. READ  emails to the desk become typed constraints
   accepted: AvailableFrom(operator='OP-SP-01', at=datetime.datetime(2026, 3, 26, 10, 0))
   accepted: DueBy(exc='EXC-10460', at=datetime.datetime(2026, 3, 26, 11, 0))
   accepted: SpeaksTo(exc='EXC-10422', lang='en')
   REJECTED time outside tonight's window: DueBy(exc='EXC-10423', at=datetime.datetime(2026, 3, 27, 9, 0))

4. PLAN  30 exceptions, 6 operators, one-hour slots from Wed 22:00 to the 15:00 cut-off
   greedy (riskiest first)  35 slots   cost    13,444   missed 4: US$3,689 at risk
   optimal (Hungarian)      35 slots   cost    12,470   missed 2: US$2,118 at risk

5. EXPLAIN  the optimal plan, for the cases a person will ask about
   EXC-10460 invoice mismatch (Citrus Paulista Exportação, pt, due 11:00) → Márcio Tavares, Thu 03:00; deadline from citruspaulista.example's email; risk US$3,473 = p 0.77 × US$4,500
   EXC-10422 wrong hs code (Citrus Paulista Exportação, en, due 12:00) → Rafaela Souza, Thu 10:00; en at brinkvos.example's request; risk US$3,033 = p 0.67 × US$4,500
   EXC-10426 wrong hs code (Vinícola Vale do Sol, en, due 11:00): NOT COVERED, its 5 qualified slots save more on other work; risk US$1,139 = p 0.25 × US$4,500
   EXC-10468 customs hold (Frutas do São Francisco, en, due 12:00): NOT COVERED, its 7 qualified slots save more on other work; risk US$979 = p 0.22 × US$4,500
   EXC-10451 missing docs (Malharia Ipê Amarelo, en, due 13:00) → João Ferreira, Thu 09:00; risk US$817 = p 0.45 × US$1,800
     (greedy left it uncovered)
   EXC-10452 invoice mismatch (Rodas & Pneus Litoral, pt, due 13:00) → Márcio Tavares, Thu 01:00; risk US$754 = p 0.42 × US$1,800
     (greedy left it uncovered)

OK: optimal is never worse than greedy, and every constraint was checked outside the solver

Forecast. Seasonal naive beats simple exponential smoothing on both lanes (MAE 3.93 against 12.00 on Santos): a flat line made on a Sunday carries the weekend's quiet into Thursday. MAPE muddles it: Jakarta has the lower MAE but the higher MAPE, because its counts are smaller, and MAPE skipped its zero day.
Score. At the same workload, logistic regression beats Rafaela's list on precision (0.68 against 0.61) and recall (0.51 against 0.46), and its Brier score beats the base rate, so its probabilities can be multiplied by money. The odds ratios read like desk knowledge (customs ×3.5, a listed shipper ×3.1, a shipper who has replied cuts the odds to a third): it agrees with Rafaela, and says by how much.
Read. Three constraints are accepted. The fourth (EXC-10423 can wait until Friday) falls outside tonight's window, so Rafaela decides. The extractor is a stand-in so the example runs offline; the validator is the real design.
Plan. On the same 35 slots, greedy leaves four exceptions uncovered and the Hungarian algorithm two, cutting the expected roll cost left from 13,444 to 12,470 invented dollars, about 7%. Both plans pass check(), which recomputes cost and constraints without trusting either planner.
Explain. Greedy spent early slots on risky exceptions that had hours of slack, then ran out of qualified operators for EXC-10451 and EXC-10452; the exact plan fits both in by moving work that could wait. Every number in an explanation is a solver input, so an LLM may rephrase the sentence but can't change the facts.

Left out on purpose: a real extraction model (the stretch exercise), side constraints across operators (the exercise), forecast uncertainty, and re-planning when an exception closes early.

In the field

Field note

Halden Freight, the week after the slice went live. Halden's VP of operations has seen the invoice-mismatch agent and wants more: "Build us an AI agent for cut-off night. The desk drowns every Thursday, containers roll, and the board has asked what we're doing with AI." The SI's account lead has already sketched an agent with tools for the queue, the roster and email, prompted to "triage and allocate the night's work".

What is actually going on. The FDE shadows two Thursdays. Of the desk's five decisions, four are numeric: staffing, priority and allocation under skills, languages, shifts and deadlines. Rafaela Souza already makes them in her head and a spreadsheet, and makes them well; containers roll because she runs out of time at 03:00, not because she chooses badly.

What the FDE does. She walks Rafaela through a one-page decision inventory first, builds the baselines from Rafaela's own list and habits, and shows each cleverer method beating them on the last eight weeks in one table. The design uses an LLM in two places, reading the inbox and drafting shipper messages; a solver makes the plan, and Rafaela approves it at 22:30 instead of building it until 03:00.

In the review.

VP of operations: "This is a forecast, a score and an optimiser. The board asked about AI. I can't take a spreadsheet model to them."

FDE: "It is AI, and the kind you can defend. Forecasting, risk scoring and optimisation have planned airline crews and supply chains for decades; the language model does what only it can do here, reading the inbox in three languages and writing to shippers. If the model allocated the night, it would cost more per run, give a different plan each time, and nobody could show the auditor why Camila got EXC-10425. This way every assignment has a reason we can print, and when the broker's limit changes we change one line. For the board: replayed nights from last quarter show fewer rolled containers and the desk finishing at 22:30."

The objection behind the objection is how it will look, so the answer gives the VP a better story: measured results and a named owner who approves every plan.

US lens. On Halden's US import leg, drayage out of Long Beach is planned against federal hours-of-service limits. Under 49 CFR 395.3, a driver of a property-carrying commercial vehicle may drive at most 11 hours within a 14-hour window that starts after 10 consecutive hours off duty, and needs a 30-minute break after 8 hours of driving (as of October 2026). For the design this is a hard constraint like Rafaela's shifts: it goes into the solver's feasibility check and into check(), never into a prompt, and the plan records which limit made a load wait. The rule has exceptions, such as the short-haul provisions in section 395.1(e), so which limits apply to which driver is the trucking company's compliance call, taken as an input, not decided by the FDE.

Decision guide

Which rung for this decision? Stop at the first row that fits.

If the decision…	Reach for	Before you climb, check
follows a written policy or threshold	a rule	who owns the rule, and where exceptions go
is "how many, which ones, how much" over data you have	a query	the definitions (Is the data ready?)
is a number for a future period	seasonal naive, then a backtested forecast	the horizon, and the error the plan can tolerate
is a yes/no or a score per row, with labelled history	the current rule, then logistic regression, then boosted trees	that every feature is known at decision time
is a plan, schedule, route or allocation under rules	a solver (assignment, MIP, CP-SAT)	every constraint written down and owned
starts from free text, documents or speech, or ends in language	an LLM with typed output and a validator	a labelled sample to measure it on
can't be written down by anyone, and a wrong answer is cheap and visible	an LLM, with a person reviewing	that "cheap and visible" is really true

Before any step uses an LLM, answer:

Is the input or the output language? If neither, why a language model?
What typed output does it return, and what validates it?
What happens when it is wrong, and how would we know?
What does it cost per run at tonight's volume and at the peak?

Misconceptions

Myth

“If we're building an agent, the LLM should make the decisions”

An agent's value is often in orchestration and language. The decisions inside it can, and often should, be made by rules, models and solvers it calls as tools.

Myth

“A trained model beats a simple rule”

Only if it does on held-out data at the workload you can staff. When the desk's rule is within a point of the model and far easier to explain, the rule ships.

Myth

“Accuracy tells me how good the classifier is”

When two thirds of exceptions resolve on their own, "nothing rolls" is 67% accurate and useless. Measure precision and recall at the cut you act on, and calibration when probabilities meet money.

Myth

“Greedy is close enough”

Sometimes, once you have measured its gap to an exact answer. On cut-off night it left twice as many exceptions uncovered, and you only know that because you solved it exactly.

Myth

“We can just tell the model the constraints”

A constraint in a prompt is a request. A constraint in a solver is satisfied by construction and proved afterwards by a checker. Shifts, skills and legal limits belong in code.

When it breaks

Symptom	Likely root cause	How you detect it	Fix / prevention
The LLM's allocation looks sensible but breaks a shift rule	constraints were put in the prompt, not enforced in code	an independent checker; operators report impossible assignments	move allocation to a solver; keep the LLM for extraction and explanation
The risk model was brilliant in testing, useless live	leakage: a feature only known after the outcome	live precision far below test; one feature dominates	audit every feature for "known at decision time"; split by time
Forecasts look fine on the dashboard but the desk is short-staffed	MAPE hid errors on small days, or the forecast ignores the weekly cycle	backtest MAE by weekday	seasonal baseline; report MAE per weekday; staff to an upper prediction bound
The solver reports "infeasible" and the plan is empty	constraints that can't all be met tonight	the solver's status; an empty plan	a "not covered" option with a cost, so the plan degrades; report which constraints bind
Plans change completely when one email arrives	near-ties in the objective	compare consecutive plans; operators complain of churn	a small cost for moving an existing assignment; re-plan only what changed
Extracted deadlines are off by hours	time zones and date formats in emails ("21/01 às 17h", "21 Jan 17:00")	per-field accuracy on a labelled sample; validator rejections	extract to a typed time with an offset; validate against the night's window; send doubtful ones to a person
A new rule takes weeks to add	rules scattered across prompts, code and spreadsheets	change lead time; the same rule found in three places	one owned list of constraints, read by the solver and the checker, with a test per rule

Exercise

Hands-on~75 min

Task. Rafaela sends two new rules on Wednesday afternoon. Add both and show what each costs. Time: 60–90 minutes.

Reefer. Any exception on a reefer container goes to an operator with the reefer skill: a late fix may mean re-plugging the box at another terminal, which only reefer-trained staff can book.
Broker capacity. Halden's customs broker at Santos takes at most five submissions tonight across the whole desk. Customs holds and HS-code corrections both count.

Starter. The names come from the worked example.

python

# Add to cutoff_night.py. A sketch: wire these in where the steps say.
BROKER_CAP = 5
BROKER_TYPES = {"CUSTOMS_HOLD", "WRONG_HS_CODE"}


def reefer_ok(e: Exc, slot: tuple[str, datetime]) -> bool:
    # e.x[3] is 1.0 for a reefer container
    return not e.x[3] or "reefer" in OPERATORS[slot[0]][2]


def broker_cases(plan: dict, excs: list[Exc]) -> int:
    return sum(1 for e in excs if e.type in BROKER_TYPES and plan[e.id] is not None)

Steps.

Before changing anything, count how many reefer exceptions each plan gives to an operator without the skill. Then add reefer_ok to feasible, so both planners and check() enforce it.
Add the broker cap to check(), so any plan that breaks it fails loudly, and run greedy unchanged.
In a comment, explain why the cap can't go into the cost matrix the way the skill rule does.
Make the exact planner respect the cap: exhaustive (try every subset of at most five broker cases allowed tonight and solve each; with nine cases that is 382 solves, about a second), penalty (add λ to every broker pair and bisect λ until at most five are covered), or OR-Tools CP-SAT (pip install ortools; one boolean per exception and slot plus one sum of at most five, as in OR-Tools' "assignment with teams" example).
Extend explain so each uncovered broker case gives its computed reason: the cap, no qualified slot, or better use elsewhere.

Done looks like.

Before the reefer rule, greedy gives 2 reefer exceptions to operators without the skill and the exact plan 5; afterwards both give 0, and check() asserts it.
Greedy covers more than five broker cases, and check() rejects its plan.
Your capped plan covers at most five and costs the same as the exhaustive search. With the published seed, the reference solution's expected cost left is 14,251 with the reefer rule alone and 15,807 with both; if yours differs, compare plans to find the rule you read differently.
The difference, tonight's cost of the broker's limit, is printed: Rafaela's case for a second broker.
The demonstration still ends with OK.

Stretch. Replace extract_constraints with a real model call that returns the same dataclasses through a JSON Schema, with the model ID read from an environment variable. Generate the Halden estate (python3 _lab/estates/halden/generate.py --out ./halden-estate), run your extractor on 20 emails from dirty/emails.jsonl, and score each field against truth/emails_truth.jsonl (deadline with offset, reply language, reefer skill). Report per-field accuracy, how many the validator rejected, and whether any rejection was in fact correct.

Check yourself

The sponsor asks for "an agent that triages and allocates the night's work". What do you write down before choosing any technique?

Simple exponential smoothing has a MAE of 12.00 on Santos and seasonal naive 3.93. Why, and what would you try next?

The model's precision is 0.68 against the list's 0.61 at the same workload. A colleague says that gain is too small to bother. What else do you look at?

Why can't the broker's five-submission limit go into the cost matrix the way "needs customs skill" does?

The extractor returns "EXC-10423 can wait until Friday 09:00" and the validator rejects it, although it only relaxes a deadline. Right call?

The VP says the board wants AI, not "a spreadsheet model". How do you answer?

Connections

Builds on: Shadowing the work as it's really done, Finding the real requirement, Baselines and ROI you can defend, The operational model: objects, links and actions
Used by: Agent or workflow? Choosing the autonomy level (once an LLM step exists, how much autonomy), Typed output: schemas, validation and tool calls (the extractor's contract), Golden datasets and the metrics that matter, Cost per resolved case, When to fine-tune, and when not to
Related: Agents over tables: text-to-SQL and beyond (the query rung), Approval gates and bounded autonomy, From requirements to eval criteria, Catching drift before the customer does

What you'll be able to do

Mental model

Core concepts

The decision inventory

The technique ladder

Baselines first

Forecasting basics

Classification on tables

Optimisation

Where LLMs are strong, and where they are weak

Hybrid patterns

Evaluating each part

Cost, explainability and change

Worked example

In the field

Decision guide

Misconceptions

When it breaks

Exercise

Check yourself

Connections

Further reading