Track 10 · Discovery & handoverLesson 02 of 12

Finding the real requirement

The stated ask is rarely the need. Learn to get underneath it.

What you'll be able to do

What you'll be able to do

01
  • You can separate a customer's stated ask from the underlying need and the success measure, and explain the difference to the person who made the ask without making them feel wrong.
  • You can run a requirements interview that collects past behaviour and evidence rather than opinions and hypotheticals.
  • You can map the stakeholders who sponsor, use, approve and block an AI deployment, and name what each one needs to see.
  • You can write a one-page problem brief with a measurable outcome, an owned success metric, sourced constraints, the non-functional requirements that kill pilots, and an assumptions log.
  • You can play the brief back to the customer, get it confirmed in writing, and trace each requirement to an eval criterion.
  • You can implement a simple linter that rejects a brief before it reaches build.
Before you start

Before you start

Starting here? On the core path this lesson follows shadowing (Shadowing the work as it's really done), which supplies its evidence; the other prerequisites can wait. Two words used below: an eval criterion is a measurable test the finished system has to pass, such as a target accuracy on a labelled sample of real cases (From requirements to eval criteria, in stage 5 of the path); an SI is the systems integrator, the consultancy that often holds the contract and runs the programme (Delivering through an SI). The examples move between Halden Freight, Northvale Bank and Carevia Health, the course's three fictional customers.

The short version

The short version

A customer asks for a thing ("build us a chatbot"). Behind the thing is a job they are trying to get done, a pain that makes the current way expensive, and a number that would tell them it got better. Your job is to find all three, write them on one page with the constraints that are genuinely fixed, and get the customer to say "yes, that is the problem" before anyone writes code. The ask is a hypothesis about a solution. The requirement is a statement about the world that you can measure.

  • Stated ask, underlying need, success measure are three different things. Most failed pilots delivered the first and were judged on the third.
  • Evidence beats opinion. Ask about the last time it happened, not what people would like. Shadowing data and event logs settle arguments that interviews start.
  • Every success metric needs a named owner, a baseline, a target and a date, or it is not a success metric.
  • Non-functional requirements kill pilots quietly. Latency, data residency, authentication and audit are found in week 1 or they are found in the security review in week 9.
  • Playback is part of the work. A brief the customer has not confirmed is your opinion, not their requirement.
Mental model

Mental model

04

Think of a doctor and a patient who arrives saying "I need antibiotics". The request is specific and confident. A good doctor does not refuse and does not comply. They ask what happened, when it started, what has been tried, and what "better" would look like. The prescription might still be antibiotics. Often it is something else. Either way, the decision now rests on a diagnosis both people agree with.

The FDE version has three layers and one gate.

Diagram
flowchart TD
    A["Stated ask<br/>'Build us a chatbot over our KYC policies'"] --> E{"Evidence<br/>shadowing, event logs,<br/>past-behaviour interviews"}
    E --> N["Underlying need<br/>Branch packs get returned;<br/>customers are contacted twice"]
    N --> M["Success measure<br/>Return rate 31% to 15% or less<br/>in 90 days, owned by Head of Branch Ops"]
    M --> B["One-page problem brief<br/>constraints, NFRs, assumptions,<br/>stakeholders"]
    B --> P{"Playback<br/>confirmed in writing?"}
    P -- no --> E
    P -- yes --> V["Eval criteria and acceptance tests<br/>(W05-01, W05-09)"]
    V --> Q["Qualification and scoping<br/>(W10-03, W10-08)"]

The loop from playback back to evidence is the important part. You will get the requirement wrong at least once. The process is designed so that you find out in a 30-minute meeting rather than at user acceptance testing.

Core concepts

Core concepts

05

Stated ask vs underlying need vs success measure

The stated ask is what the customer says they want. It is usually phrased as a solution ("a chatbot", "an agent that handles all exceptions") because solutions are easier to talk about than problems, and because a board, a competitor demo or an analyst report suggested it.

The underlying need is the job the organisation is trying to get done and why the current way is painful, in operational language: cases returned, calls repeated, claims pended, shipments held.

The success measure is the observable number that would move if the need were met, with a baseline, target, time window, data source and owner.

The three often point in different directions. A sponsor asking for a KYC policy chatbot might be judged next quarter on how many refresh packs central operations returns. A chatbot that answers policy questions well could leave that number untouched, because the officers already know the policy and the errors come from missing scans and a form default. You can build the ask perfectly and fail the measure completely.

Write all three down separately. When they disagree, the success measure wins, because it is what the customer will use to judge you.

The "build me a chatbot" problem

"Build me a chatbot" is a very common stated ask, and hard to push back on because it is concrete, cheap-sounding and easy to demo. It causes three predictable problems.

  1. It fixes the interface before the job is known. Chat assumes a person will think of a question, type it and read an answer. Many jobs are better served by a check that runs on submission, a pre-filled form or a ranked worklist. Chat fits varied, unpredictable questions, not the same five checks every time.
  2. It hides the success measure. Usage becomes the metric by default, and usage is not value. Staff can use a chatbot heavily because the intranet search is worse, while the business outcome does not move.
  3. It invites unbounded scope. "A chatbot over our policies" grows to all 4,000 policies, all branches, all languages. Nothing in the ask says where to stop.

The move is not to refuse. It is to accept the ask as a hypothesis and ask what it is for: "If the chatbot worked perfectly, what would be different in three months? What number would your boss look at?" Sometimes the answer confirms chat is right. More often it reveals a narrower, more measurable job that an agent, a workflow or no AI at all handles better (see Agent or workflow? Choosing the autonomy level).

Jobs-to-be-Done framing

Jobs-to-be-Done (JTBD), popularised by Clayton Christensen and colleagues, says customers "hire" a product to make progress in a particular circumstance. The unit of analysis is the job and its circumstance, not the product or the customer demographic. The framing is useful to FDEs because it pulls the conversation away from features.

A job statement has three parts:

When [circumstance], I want to [make this progress], so that [outcome that matters to me].

For a Northvale branch officer: "When a customer is at my desk for their periodic KYC refresh, I want to know before I submit that the pack is complete for their segment, so that I do not have to call them back next week and explain why we need another document."

Three things fall out of a good job statement:

  • The circumstance sets non-functional requirements. "Customer at my desk" means latency matters in seconds, not minutes.
  • The progress sets the functional scope. "Know the pack is complete" is a checking job, not a question-answering job.
  • The outcome sets the success measure. "Not calling them back" maps directly to the return rate.

JTBD was built for product innovation, so it under-weights organisational jobs no individual feels, such as audit readiness. Write jobs at two levels: the individual (the officer) and the organisation (refreshes completed by their due date). When they conflict ("get the customer out fast" versus "collect every document"), you have found a design constraint, not a contradiction to paper over.

Five whys and its limits

Five whys is the practice of asking "why" repeatedly until you get past symptoms to a cause. It comes from the Toyota Production System. The Lean Enterprise Institute's description makes the point that five is not magic: you keep asking until you reach the root cause.

Applied to Northvale:

  1. Why do branches want a policy chatbot? Because officers keep getting KYC packs returned.
  2. Why are packs returned? Because documents are missing or wrong for the customer's segment.
  3. Why are they wrong? Because officers use a printed checklist that does not distinguish NRI customers from resident customers.
  4. Why does it not distinguish them? Because the checklist was last updated before the 2024 policy change.
  5. Why was it not updated? Because no one owns the branch checklist since the policy team was reorganised.

That last answer reveals an ownership gap that no chatbot will fix. It also suggests the fastest win may be updating a checklist, which is a useful thing to know before you spend six weeks building.

Five whys has real limits:

  • It follows one causal chain. Most operational problems have several causes. Here, missing scans and a form default also contribute; a single chain hides them.
  • It stops where the asker's knowledge stops. Ask three people and you often get three chains.
  • It drifts towards blame. "Because the officer was careless" is rarely the actionable cause.
  • It produces stories, not frequencies. It tells you a cause exists, not how much of the problem it explains.

Use five whys to generate hypotheses, then test them against data. The return-reason codes tell you what share of returns each cause explains. If a cause explains 8% of returns, it is not the requirement, however vivid the story.

Problem statements with measurable outcomes

A problem statement describes the current state, who suffers it, how often, and what it costs. It does not name a solution. A measurable outcome describes the change you are aiming for in numbers.

A weak statement: "Branch staff need a better way to find KYC policy information."

A strong statement: "Periodic KYC refresh packs submitted by 12 pilot branches are returned by central KYC operations for document errors in 31% of cases (April to June, n = 4,180). Each return adds a second customer contact and pushes the refresh past its due date in most cases. The bank wants returns for document errors at or below 15% within 90 days of go-live, measured from the case system's return-reason codes, owned by the Head of Branch Operations."

The outcome has six parts. If any is missing, the outcome is not yet measurable:

PartQuestion it answersExample
MetricWhat exactly is counted?Share of refresh packs returned for document errors
BaselineWhat is it now, over what sample?31%, April to June, 12 branches, n = 4,180
TargetWhat number counts as success?15% or less
Data sourceWhere does the number come from?Return-reason codes D01 to D07 in the KYC case system
Time windowBy when, and over what period?90 days after go-live, rolling 30-day measurement
OwnerWho is accountable for the number and will defend it?Head of Branch Operations

The baseline is often missing on day one. That is fine as long as measuring it becomes the first task. Baselines and ROI you can defend covers how to measure it properly. What is not fine is a target without a baseline, because "reduce returns by half" means nothing if nobody knows the current rate.

Also write down guardrail outcomes: things that must not get worse. If returns drop because officers submit fewer refreshes, you have not succeeded. For Northvale: refresh completion volume must not fall, and average branch handling time must not rise by more than 2 minutes per pack.

Stakeholder mapping

Enterprise AI projects often die not because the sponsor loses interest but because someone with a veto was never asked. Map four roles before you write the brief.

RoleWho they usually areWhat they need from youWhat happens if you miss them
SponsorBusiness leader who owns the budget and the outcomeA date, a measurable result, a story for their bossNo budget for phase 2
UsersThe people whose work changes: officers, agents, analysts, ops staffLess rework, no extra clicks, trust that they will not be blamed for the tool's errorsLow adoption, quiet workarounds, the tool is used for the demo and then ignored
ApproversModel risk, CISO / security architecture, data protection officer, compliance, change advisory boardEvidence in their format: threat model, data flow, model inventory entry, test resultsGo-live blocked in the last fortnight
BlockersAnyone whose interests are threatened or whose consent is legally required: IT platform owners, legal, procurement, works councils or employee representatives, the SI's delivery leadEarly notice, a seat at the table, a clear scope boundaryEscalations, delays, or a formal objection that stops the rollout

The less obvious ones:

  • Model risk. In regulated firms, a tool that informs a regulated decision (credit, KYC, claims) often counts as a model under the firm's model risk policy and needs validation. Ask in week 1 how the policy classifies your use case (frameworks are in Governance built in, not bolted on).
  • Legal, data protection and IT. They decide what data the system may see and own the systems you read from. Access tickets often take longer than the build.
  • Works councils and employee representatives. A system that logs which employee did what can count as performance monitoring, and in some jurisdictions staff representatives have co-determination rights over it. Shadowing the work as it's really done covers the German example; ask HR or employee relations before you design per-user logging.

For each stakeholder, record their interest in the outcome, their power to stop it, and the evidence that will satisfy them. The output is a list of meetings to hold in the next two weeks.

Interview technique: past behaviour, not hypotheticals

Most requirement interviews collect opinions. People are polite, optimistic and bad at predicting their own behaviour. Ask "Would a tool that checks the pack before submission be useful?" and almost everyone says yes. That answer tells you nothing.

Rob Fitzpatrick's The Mom Test gives three rules for conversations that produce usable evidence even when people want to be nice to you. Paraphrased: talk about their life rather than your idea; ask about specific things that happened in the past rather than generic or future opinions; and talk less, listen more. The book is written for founders, but the rules transfer directly to enterprise discovery.

Practical moves:

Instead ofAskWhy
"Would this be useful?""Tell me about the last pack that was returned. What happened?"A real case has details you can verify
"How long does a refresh take?""Walk me through yesterday's refreshes. Which took longest?"Specific recall beats an averaged estimate
"What features do you want?""What do you do today when you are unsure which documents apply?"Reveals workarounds and the real job
"Is accuracy important?""What happens when the checklist is wrong? Who finds out, and when?"Reveals the cost of errors, which sets the accuracy bar
"Would you use it?""What have you tried before to fix this? Why did it stop?"Past attempts show both commitment and failure modes

Other habits that matter:

  • Open questions first, then narrow. "Talk me through", "what happened next". Close with specific confirmations.
  • Ask for the artefact. "Can you show me the returned pack?" Artefacts are evidence; descriptions are memory.
  • Interview deciders and doers separately. The sponsor knows the outcome they are judged on. Users know how the work is done.
  • Listen for workarounds and do not pitch. Private spreadsheets and "I just call Priya in central ops" signal an unmet need. Describing your solution turns the interview into a sales call; save it for playback.

Pair interviews with shadowing (concept 01). When an interview and the event log disagree, the log is usually closer to the truth, and the gap is often the requirement.

Separating constraints from preferences

Customers state many things as if they were fixed. Some are. Many are preferences, assumptions or habits wearing the clothes of constraints. The difference matters because constraints shape the design, while preferences are negotiable trade-offs.

A constraint has a source you can point to: a regulation, a written policy, a contract, an architecture standard, a budget line. A preference is someone's wish, however senior.

StatementConstraint or preference?Test
"Indian customer data stays in India"Constraint (data localisation obligation)Ask for the policy reference; involve the DPO
"It must run on Azure"Usually a preference, sometimes an architecture standardAsk whether there is an approved-cloud standard and what the exception process is
"We need it in four weeks"Often a preference tied to a board dateAsk what happens on that date and what must be shown
"No changes to the core banking system"Constraint (change freeze, mainframe change cost)Confirm with IT; ask when the next window is
"It has to use our existing chatbot vendor"Could be a contract (licence) or a preferenceAsk procurement whether a contract commits volume
"Answers must be 100% accurate"Neither; a fearAsk what the current error rate is and what an error costs

In the brief, label each constraint with its source. Anything without a source is recorded as a preference. You still honour preferences where you can; you just know which ones you can trade when the timeline gets tight.

Non-functional requirements that kill pilots

Functional requirements decide whether the demo impresses. Non-functional requirements (NFRs) decide whether it goes live. Four recur in AI pilot reviews, and all four are cheap to ask about early and expensive to discover late.

NFRThe question to ask in week 1Why it kills pilotsWhere it is taught in depth
Latency"What is the person doing while they wait? How long before they give up?"A 20-second agent is fine for a back-office batch and unusable with a customer at the deskSLOs and latency budgets
Data residency"Which data, which jurisdictions, where may it be processed and stored, including by model providers?"The model region or a log sink outside the permitted region fails reviewData residency and sovereignty
Authentication and authorisation"Whose identity does the system act under? What may each user see?"A shared service account that can read all customers fails the CISO reviewSSO and provisioning for agents
Audit"Who needs to reconstruct what the system did, for which decision, how long after?"No record of what the tool showed, to whom, with which policy version, means no defensible decision trailThe audit pack

Others matter too: availability windows, peak volume, languages, accessibility, retention, cost per transaction, and the change-window calendar. Ask about all of them, but never leave the room without the first four. Write latency as a number with a percentile ("6 s p95"), not as an adjective.

Writing it down: the one-page problem brief and the assumptions log

The brief is the artefact. Keep it to one page so it is read:

  1. Problem statement (no solution words) and job statements (individual and organisational).
  2. Outcomes: primary metric with all six parts; one or two guardrails.
  3. Stakeholders with what each needs to see.
  4. In / out of scope at the level of the job (slicing comes later, in Scoping a shippable slice).
  5. Constraints with sources; preferences labelled; NFRs.
  6. Evidence with dates, assumptions (link to the log) and the playback record.

The assumptions log is a separate, living list: the assumption, why you believe it, what breaks if it is wrong, an owner who can confirm it, and a validate-by date. "The SharePoint policy library is current" is an assumption; if it is wrong, every citation is wrong. Review the log at every check-in. An assumption that passes its date unchecked becomes a risk on the risk register.

Confirming playback with the customer

Playback is a short meeting, 20 to 30 minutes, where you read the brief back to the sponsor and one or two key users and ask them to correct it. It is not a presentation of your solution.

How to run it:

  • Lead with evidence and name the gap. "Of 4,180 packs, 1,296 were returned. Three reasons explain most of them. You asked for a policy chatbot; we think the real problem is pack completeness at submission. Is that right?"
  • Ask for corrections, not approval. "What have we got wrong?" beats "Does this look good?"
  • Confirm the metric owner in the room. No owner means no definition of success.
  • Record it in writing the same day, with the brief attached, and ask for a one-line reply. In an SI-led engagement, route it through the partner's account lead so it lands in their change-control record (Delivering through an SI).

Expect playback to change something. If nothing changes, ask the sponsor to restate the problem in their own words before you trust it.

From requirement to eval criteria

Every line in the brief should be testable later. From requirements to eval criteria owns this translation in depth: it turns each requirement into a metric, a threshold, a test set and a method. In discovery, you only need to make sure each requirement has an eval hook: a note on how it will be tested and against what data. "Flag every mandatory missing document" has a hook: recall against a golden set of returned packs. "Answers should be helpful" has none, and should not survive into the brief. The acceptance criteria that the customer finally signs (Acceptance criteria and sign-off) are built from these hooks, so a requirement without one becomes an argument at sign-off.

Worked example

Worked example

06

You will build a linter for problem briefs. It takes a structured brief (here, JSON, so it runs with the standard library; the same shape works as YAML if you add PyYAML) and flags the gaps that most often cause arguments later:

  • a problem statement that names a solution;
  • outcomes without a metric, baseline, target, data source or deadline, or with non-numeric targets;
  • success metrics with no named owner, or an owner who is not in the stakeholder map;
  • untestable adjectives ("fast", "accurate", "seamless") with no threshold;
  • requirements with no eval hook;
  • missing latency, residency, auth or audit NFRs, and latency without a number;
  • constraints with no source (possibly preferences);
  • assumptions with no owner or validate-by date;
  • a brief that has not been played back and confirmed.

Requirements: Python 3.11+, standard library only. Runs offline.

python
"""Problem-brief linter: flags the gaps that turn a pilot into an argument.

Runs offline with the standard library only. Python 3.11+.
"""
from __future__ import annotations

import json
import re
import sys
from dataclasses import dataclass
from typing import Any

# Phrases that put a solution inside the problem statement.
SOLUTION_PHRASES = (
    "chatbot", "copilot", "agent", "llm", "genai", "gen ai", "rag",
    "dashboard", "portal", "machine learning", "automate with", "build a", "build an",
)
# Adjectives that cannot be tested unless a number travels with them.
UNTESTABLE = (
    "fast", "quick", "quickly", "accurate", "seamless", "intuitive", "user-friendly",
    "robust", "scalable", "real-time", "easy", "secure", "reliable", "efficient",
)
# Non-functional requirements that most often kill enterprise AI pilots.
REQUIRED_NFRS = ("latency", "residency", "auth", "audit")
NFR_NEEDS_NUMBER = ("latency",)
CONSTRAINT_SOURCES = ("regulation", "policy", "contract", "architecture-standard", "budget")
PREFERENCE = "preference"  # an explicitly labelled preference is fine: it is negotiable
HAS_NUMBER = re.compile(r"\d")


@dataclass(frozen=True)
class Finding:
    severity: str  # "ERROR" blocks sign-off; "WARN" needs a decision
    code: str
    where: str
    message: str


def _words(text: str) -> set[str]:
    return set(re.findall(r"[a-z][a-z\-]*", text.lower()))


def _check_problem(brief: dict[str, Any]) -> list[Finding]:
    out: list[Finding] = []
    text = str(brief.get("problem_statement", "")).strip()
    if not text:
        return [Finding("ERROR", "P001", "problem_statement", "missing problem statement")]
    lowered = f" {text.lower()} "
    for phrase in SOLUTION_PHRASES:
        if re.search(rf"\b{re.escape(phrase)}\b", lowered):
            out.append(Finding("ERROR", "P002", "problem_statement",
                               f"names a solution ('{phrase}'); describe the job and the pain instead"))
    return out


def _check_outcomes(brief: dict[str, Any], stakeholders: set[str]) -> list[Finding]:
    out: list[Finding] = []
    outcomes = brief.get("outcomes") or []
    if not outcomes:
        return [Finding("ERROR", "O001", "outcomes", "no measurable outcome defined")]
    for i, o in enumerate(outcomes):
        where = f"outcomes[{i}]"
        for field in ("metric", "baseline", "target", "measured_by", "by_when"):
            if not str(o.get(field, "")).strip():
                out.append(Finding("ERROR", "O002", where, f"missing '{field}'"))
        for field in ("baseline", "target"):
            value = str(o.get(field, ""))
            if value and not HAS_NUMBER.search(value):
                out.append(Finding("ERROR", "O003", where, f"'{field}' has no number: '{value}'"))
        owner = str(o.get("owner", "")).strip()
        if not owner:
            out.append(Finding("ERROR", "O004", where, "success metric has no named owner"))
        elif owner not in stakeholders:
            out.append(Finding("WARN", "O005", where, f"owner '{owner}' is not in the stakeholder map"))
    return out


def _check_adjectives(brief: dict[str, Any]) -> list[Finding]:
    out: list[Finding] = []
    for section in ("requirements", "nfrs"):
        for i, item in enumerate(brief.get(section) or []):
            text = str(item.get("text", ""))
            hits = sorted(_words(text) & set(UNTESTABLE))
            if hits and not HAS_NUMBER.search(text):
                out.append(Finding("ERROR", "R001", f"{section}[{i}]",
                                   f"untestable adjective(s) {hits} with no threshold"))
            if section == "requirements" and not item.get("eval_hook"):
                out.append(Finding("WARN", "R002", f"{section}[{i}]",
                                   "no eval hook: how will this be tested? (see W05-01)"))
    return out


def _check_nfrs(brief: dict[str, Any]) -> list[Finding]:
    out: list[Finding] = []
    by_cat = {str(n.get("category", "")).lower(): n for n in brief.get("nfrs") or []}
    for cat in REQUIRED_NFRS:
        nfr = by_cat.get(cat)
        if nfr is None:
            out.append(Finding("ERROR", "N001", "nfrs", f"missing non-functional requirement: {cat}"))
        elif cat in NFR_NEEDS_NUMBER and not HAS_NUMBER.search(str(nfr.get("text", ""))):
            out.append(Finding("ERROR", "N002", f"nfrs[{cat}]", f"{cat} stated without a number"))
    return out


def _check_constraints(brief: dict[str, Any]) -> list[Finding]:
    out: list[Finding] = []
    for i, c in enumerate(brief.get("constraints") or []):
        source = str(c.get("source", "")).lower()
        if source not in CONSTRAINT_SOURCES and source != PREFERENCE:
            out.append(Finding("WARN", "C001", f"constraints[{i}]",
                               f"'{c.get('text', '')}' has no source; hard constraint or preference?"))
    return out


def _check_assumptions_and_playback(brief: dict[str, Any]) -> list[Finding]:
    out: list[Finding] = []
    for i, a in enumerate(brief.get("assumptions") or []):
        if not a.get("owner") or not a.get("validate_by"):
            out.append(Finding("WARN", "A001", f"assumptions[{i}]",
                               "assumption needs an owner and a validate-by date"))
    playback = brief.get("playback") or {}
    if not playback.get("confirmed_by") or not playback.get("date"):
        out.append(Finding("ERROR", "B001", "playback", "brief has not been played back and confirmed"))
    return out


def lint(brief: dict[str, Any]) -> list[Finding]:
    stakeholders = {str(s.get("name", "")) for s in brief.get("stakeholders") or []}
    findings: list[Finding] = []
    findings += _check_problem(brief)
    findings += _check_outcomes(brief, stakeholders)
    findings += _check_adjectives(brief)
    findings += _check_nfrs(brief)
    findings += _check_constraints(brief)
    findings += _check_assumptions_and_playback(brief)
    return findings


def report(name: str, brief: dict[str, Any]) -> int:
    findings = lint(brief)
    errors = sum(f.severity == "ERROR" for f in findings)
    print(f"== {name}: {errors} error(s), {len(findings) - errors} warning(s)")
    for f in findings:
        print(f"  {f.severity:<5} {f.code} {f.where:<16} {f.message}")
    return errors


AS_RECEIVED = json.loads("""{
  "problem_statement": "Build a GenAI chatbot over our KYC policy documents for branch staff.",
  "outcomes": [{"metric": "staff productivity", "target": "significantly better"}],
  "stakeholders": [{"name": "Head of Retail Operations", "role": "sponsor"}],
  "requirements": [{"text": "Answers must be accurate and fast."}],
  "nfrs": [{"category": "auth", "text": "Use Okta SSO."}],
  "constraints": [{"text": "Must run on Azure", "source": ""}],
  "assumptions": [{"text": "Policies are up to date in SharePoint"}],
  "playback": {}
}""")

REVISED = json.loads("""{
  "problem_statement": "Periodic KYC refresh packs submitted by branches are returned by central KYC operations for missing or wrong documents, so customers are contacted twice and refreshes miss their due date.",
  "outcomes": [{
    "metric": "share of KYC refresh packs returned by central ops for document errors",
    "baseline": "31% (Apr-Jun, 12 pilot branches, n=4,180)",
    "target": "<=15%",
    "measured_by": "KYC case system return-reason codes D01-D07",
    "by_when": "90 days after go-live",
    "owner": "Head of Branch Operations"
  }],
  "stakeholders": [
    {"name": "Head of Retail Operations", "role": "sponsor"},
    {"name": "Head of Branch Operations", "role": "metric owner"},
    {"name": "Model Risk Management", "role": "approver"},
    {"name": "CISO office", "role": "approver"},
    {"name": "Employee Relations", "role": "possible blocker"}
  ],
  "requirements": [
    {"text": "Flag every mandatory document missing for the customer segment before submission; recall >=95% on the golden set of 300 returned packs.",
     "eval_hook": "golden set recall, W05-02"},
    {"text": "Cite the policy clause for every flag.", "eval_hook": "citation presence 100%, clause match >=90%"}
  ],
  "nfrs": [
    {"category": "latency", "text": "Checklist result in <=6 s p95 while the customer is at the desk."},
    {"category": "residency", "text": "Indian customer data processed and stored in India-region infrastructure only."},
    {"category": "auth", "text": "Okta SSO; agent acts on behalf of the signed-in officer; no shared service login to the KYC system."},
    {"category": "audit", "text": "Log officer ID, pack ID, policy version, flags shown and override reason; retain per records policy."}
  ],
  "constraints": [
    {"text": "Indian customer data stays in India", "source": "regulation"},
    {"text": "Production changes only in the Saturday change window", "source": "policy"},
    {"text": "Prefer Azure OpenAI", "source": "preference"}
  ],
  "assumptions": [
    {"text": "SharePoint policy library is the current version", "owner": "KYC Policy team", "validate_by": "2026-10-09"}
  ],
  "playback": {"confirmed_by": "Head of Retail Operations", "date": "2026-10-02"}
}""")

if __name__ == "__main__":
    total = report("as received", AS_RECEIVED)
    print()
    total_revised = report("revised", REVISED)
    sys.exit(1 if total_revised else 0)

Run it with python3 brief_lint.py. Output:

text
== as received: 13 error(s), 3 warning(s)
  ERROR P002 problem_statement names a solution ('chatbot'); describe the job and the pain instead
  ERROR P002 problem_statement names a solution ('genai'); describe the job and the pain instead
  ERROR P002 problem_statement names a solution ('build a'); describe the job and the pain instead
  ERROR O002 outcomes[0]      missing 'baseline'
  ERROR O002 outcomes[0]      missing 'measured_by'
  ERROR O002 outcomes[0]      missing 'by_when'
  ERROR O003 outcomes[0]      'target' has no number: 'significantly better'
  ERROR O004 outcomes[0]      success metric has no named owner
  ERROR R001 requirements[0]  untestable adjective(s) ['accurate', 'fast'] with no threshold
  WARN  R002 requirements[0]  no eval hook: how will this be tested? (see W05-01)
  ERROR N001 nfrs             missing non-functional requirement: latency
  ERROR N001 nfrs             missing non-functional requirement: residency
  ERROR N001 nfrs             missing non-functional requirement: audit
  WARN  C001 constraints[0]   'Must run on Azure' has no source; hard constraint or preference?
  WARN  A001 assumptions[0]   assumption needs an owner and a validate-by date
  ERROR B001 playback         brief has not been played back and confirmed

== revised: 0 error(s), 0 warning(s)

The process exits 0 because the revised brief is clean. Wire lint() into a pre-commit hook or a CI job on the repository where briefs live, and a brief cannot move to "ready for build" with errors.

What the output teaches:

  • The as-received brief is typical, not a straw man. Solution words, "significantly better" and "accurate and fast" are what most first-week briefs look like.
  • The revised brief changed the problem, not just the wording. It moved from "policy questions" to "pack completeness", and that shift came from the return-reason data.
  • "Prefer Azure OpenAI" survives as a labelled preference. The linter does not ban preferences; it insists you know which statements are negotiable.
  • A clean lint is necessary, not sufficient. It cannot tell whether 15% is the right target or whether the owner really agreed. That is what playback is for.

The heuristics are deliberately crude: a problem about travel agents would trip the "agent" rule. Treat the linter as a reviewer that asks questions, and add an override field with a reason for false positives.

In the field

In the field

07
Field note

The ask. Northvale Bank's Head of Retail Operations emails the SI partner's account lead: "We want a GenAI chatbot for branch staff over our KYC policies. Competitors have one. We need it live in the 12 pilot branches before the quarterly review in eight weeks." The SI forwards it to you with a draft statement of work that says "policy Q&A chatbot".

What is actually going on. Two half-days shadowing in branches and an afternoon with central KYC operations in Mumbai show that officers rarely look up policy; they know it. They work from a laminated checklist, scan on a shared scanner that drops pages, and submit through a form that defaults every customer to "resident individual". The case system's return-reason data shows three causes explain most returns: missing address proof for NRI customers, expired documents and incomplete scans. Officers also keep a WhatsApp group with a central ops checker who answers "is this enough?" informally, the clearest sign of an unmet need.

What the FDE does.

  1. Writes the three layers: the ask (policy chatbot), the need (packs complete at submission, customer contacted once), the measure (document-error return rate, baseline 31%).
  2. Maps stakeholders. The sponsor owns the quarterly review; the Head of Branch Operations owns the return rate; model risk will ask whether the tool influences a KYC decision; the CISO office will ask whose identity reads the case system; employee relations needs to know the audit log records officer IDs.
  3. Separates constraints from preferences. India-only processing of Indian customer data has a policy reference from the DPO. "Azure" came from the sponsor's last vendor meeting; the architecture standard also allows the on-prem platform.
  4. Captures the NFRs (the four shown in the revised brief above), writes the brief and assumptions log, lints it, and books a 30-minute playback with the sponsor and two branch managers.

The playback, and the objection. The sponsor's first reaction: "I asked for a chatbot. The board has seen chatbots. Why are you telling me about return rates?"

The answer that works is to give the sponsor a better story for their board, not a lecture: "You can still show the board an assistant. What we are proposing is that it does the one job that moves the number you will be asked about. If returns drop from 31% to 15%, that is roughly 670 fewer repeat customer contacts a quarter in the pilot branches alone, using the April to June volume. A policy chatbot would be harder to tie to any number. We can add a question-answering mode in phase 2 once the checker has earned trust, and the policy index we build for citations is the same one it would need."

A second objection, from model risk. "If the tool tells an officer a pack is complete, it is influencing a KYC decision. Under our policy that makes it a model. It needs validation." The FDE does not argue the classification. They ask for the policy's tiering criteria, propose that the tool only flags missing items and never marks a pack as approved, keep the central ops checker as the decision maker, and offer the golden-set evaluation as validation evidence. That keeps the system in the "agent proposes, human signs" shape and gives model risk something concrete to review.

What changed after playback. The sponsor accepted the measure and named the Head of Branch Operations as owner. A branch manager noted the NRI problem is concentrated in four branches, which narrowed the first slice. "Eight weeks" turned out to mean "show progress at the quarterly review", not "fully live". All three changes went into the brief, the SI's change log and the same-day playback email.

Decision guide

Decision guide

08

Use this when you are handed an ask.

Signal in the ask or interviewWhat it usually meansWhat you do next
The ask names a technology ("chatbot", "agent", "GenAI")Solution chosen before the problemAsk "If it worked perfectly, what number would change, and who would notice?"
The success measure is usage, adoption or satisfaction onlyNo business outcome agreedFind the operational metric the sponsor is judged on; keep usage as a secondary signal
Nobody will own the metricNo definition of successEscalate to the sponsor before building; do not proceed without an owner
Interviews and event logs disagreeThe documented process is not the real oneTrust the log, investigate the gap; the gap is often the requirement
A "constraint" has no sourceProbably a preferenceAsk for the policy, contract or standard; label it a preference if none
Latency, residency, auth or audit not discussedPilot will fail review lateAsk the four NFR questions before the brief is written
Users have private spreadsheets, WhatsApp groups, sticky notesUnmet need with a workaroundStudy the workaround; it often defines the minimum viable feature
The sponsor and users describe different problemsTwo jobs, possibly in conflictWrite both job statements; resolve the conflict in playback, not in code

Exit test for discovery: you can state the job in the user's words, the cost of failure today, the one metric with baseline, owner and date, who can stop the project, which constraints have sources, the four NFRs in numbers, and the open assumptions with owners. And the customer has confirmed it in writing.

Misconceptions

Misconceptions

09

Myth

“The customer knows what they want; my job is to build it”

The customer knows what hurts and what they will be judged on. They rarely know what an AI system can and cannot do reliably, so their proposed solution is a guess. Your job is to combine their knowledge of the problem with your knowledge of the solution space, and that means questioning the ask.

Myth

“Pushing back on the ask will damage the relationship”

Building the wrong thing damages it far more, and later. Sponsors rarely object to being asked what success looks like. They object to being told they are wrong. Frame the conversation around their success measure and offer a better story for their boss.

Myth

“Non-functional requirements are for the architecture phase”

Residency, auth and audit decide which platforms, model endpoints and integration patterns are even possible. Discover them late and you rebuild. They belong in the first week's brief.

Myth

“Five whys finds the root cause”

It finds a root cause along one chain, as seen by one person. Use it to generate hypotheses, then use data to find out how much each cause explains.

Myth

“A signed SOW is the requirement”

A statement of work is often written before discovery and describes the stated ask. Treat it as a constraint on scope and budget, and route requirement changes through change control.

When it breaks

When it breaks

10
SymptomLikely root causeHow you detect itFix / prevention
Pilot is "used a lot" but the sponsor says it did not deliverBuilt to the stated ask; success measure never agreedNo baseline or owner for a business metric in the briefAgree the measure and owner before build; lint briefs for owned outcomes
Go-live blocked two weeks before the date by security or model riskApprovers not mapped; NFRs not capturedApprover names absent from stakeholder map; no threat model or model-inventory entry scheduledMap approvers in week 1; capture the four NFRs; book the reviews early
Users bypass the tool and keep the old workaroundTool solves the documented job, not the real oneLow repeat usage; the WhatsApp group or spreadsheet is still activeShadow before writing requirements; design around the workaround
Endless scope growth during buildProblem statement named a solution with no boundary"All policies", "all branches", "all languages" in scope; no in/out listWrite the job, not the feature; define out-of-scope explicitly; go to concept 08
Sign-off argument about "accuracy"Untestable adjectives in requirementsRequirements with "accurate", "fast" and no threshold or eval hookReplace adjectives with numbers and test methods during brief review
A critical assumption turns out false mid-build (stale policy library, missing data)Assumption never logged or never checkedAssumptions without owners or dates; no review cadenceKeep an assumptions log with owners and validate-by dates; review weekly
Customer says "that is not what we agreed"No written playbackNo confirmation email or change-log entryPlay back, confirm in writing, route changes through the SI's change control
Exercise

Exercise

11
Hands-on~60 min

Scenario. Carevia Health's VP of Utilisation Management sends this ask: "We want an AI agent that approves prior authorisation requests automatically so our nurses can focus on complex cases. It needs to be fast and accurate. Our Salesforce Health Cloud team can support. Target is go-live in Q1."

You also have these discovery notes (fictional):

  • The prior-auth queue receives about 9,000 requests a week. Around 40% are missing clinical documentation on arrival and are pended for outreach.
  • Nurses say "most of my day is chasing faxes and calling provider offices".
  • Coverage determinations must remain human decisions (Carevia's policy; see the shared-universe constraints).
  • Turnaround-time reporting is tracked by the Utilisation Management operations analyst.
  • PHI must stay within the HIPAA-covered environment; the CISO requires SSO through the existing identity provider.

Steps (60–90 minutes).

  1. Write the three layers: stated ask, underlying need, success measure. Expect the need to differ from the ask.
  2. Write an individual job statement for a UM nurse and an organisational job statement for Carevia.
  3. Draft three past-behaviour interview questions for nurses and two for the provider-relations team. No hypotheticals.
  4. Build a stakeholder map with at least one sponsor, one user group, two approvers and one blocker.
  5. Write the brief as JSON in the linter's shape. Include a primary outcome with all six parts (invent a plausible baseline and label it "to be measured"), one guardrail outcome, four NFRs, at least three constraints labelled with sources or as preferences, and two assumptions.
  6. Run the linter. Fix every ERROR. Decide whether each WARN needs a fix or an override with a reason.
  7. Write a five-sentence playback script for the VP, including how you will handle "I asked for automatic approvals".

Done looks like.

  • The problem statement contains no solution words and the linter reports 0 errors.
  • The primary outcome has a metric, baseline, target, data source, deadline and an owner who appears in the stakeholder map.
  • The brief states explicitly that coverage determinations stay human, and the requirements describe what the system prepares or proposes for the nurse, not what it decides.
  • Each requirement has an eval hook a reviewer could turn into a test.
  • The playback script names the gap between the ask and the need in one sentence.

Stretch. Extend the linter with two rules: (1) flag any outcome whose by_when is not an ISO date or a relative period like "90 days after go-live"; (2) flag a brief where any requirement implies an autonomous consequential decision ("approve", "deny", "decide") without a matching human-approval requirement. Add a test brief that trips each rule.

Check yourself

Check yourself

12
1

A sponsor says, "Success is that staff love using it." Why is this not a success measure, and what would you ask next?

2

Rewrite this interview question so it collects evidence: "Would an assistant that drafts customer replies save you time?"

3

Your five-whys chain ends with "because the officer did not read the policy". What is wrong, and what do you do?

4

The customer says, "It must run on Azure." How do you decide whether this is a constraint or a preference, and why does it matter?

5

Name the four non-functional requirements this concept says kill pilots, and give the week-1 question for each.

6

Playback with the sponsor goes smoothly and nothing changes. Should you be pleased?

Connections

Connections

13
Further reading

Further reading

14