Work on your own machine.

python3 _lab/estates/halden/generate.py --out ./halden-estate
python3 _lab/estates/halden/generate.py --out ./halden-estate --sites all          # add Nhava Sheva and the US leg
python3 _lab/estates/halden/generate.py --out ./halden-estate --schema v2          # the SAP extract after the change
python3 _lab/estates/halden/generate.py --self-test                                # about 7 s
python3 _lab/estates/halden/generate.py --widget-sample _lab/widgets/halden-queue-v1.json

Widget sample. _lab/widgets/halden-queue-v1.json feeds the operator-workbench widget in W09-11: twelve São Paulo exceptions open at 09:00 on Thursday 2 April, six hours before the Santos cut-off, each with the agent's proposal, its evidence and the desk lead's decision. Eight proposals are right. Four carry a problem an operator should catch: a wrong desk label, an invoice amount misread from Portuguese number format (D08), the wrong document requested, and a desk rule the agent doesn't know. Ids, amounts and documents come from the estate, so the widget and the lab agree. The site copies it to content/widgets/.

The operation

• Exceptions pile up before cut-off: in the last 6 hours they arrive at about three times the rate of the day before.
• About 5% of bookings are rolled to the next sailing, 15% leave one to three days late, and 22% arrive late.
• The desk's label is wrong on about 8% of exceptions (a rushed operator picks a neighbouring type); the true type is in truth/exceptions_truth.csv. Shippers on the desk's known-bad list really do raise three times as many exceptions.
• The US leg adds importer security filings (19 CFR 149.2 sets 24 hours before lading at the foreign port for most elements, as of October 2026; the estate applies it to the whole filing, and about 8% are late), drayage appointments at the port (about 7% missed) and carrier check calls.
• The core estate is about 5,800 shipments, 27,000 carrier events, 4,900 exceptions, 6,900 documents and 300 emails; --sites all adds roughly a third.

Files

Both clean/ and dirty/ hold the same files. Line numbers in truth/defects.json are 1-based and count the CSV header as a line.

Columns

sap_shipments.csv (schema v1)

Schema v2 (--schema v2), the change the residency inflicts in week 5. Only sap_shipments.csv changes:

carrier_events.csv: carrier_name (as the portal prints it), container_no, event_code (GATE_IN, LOADED, DEPARTED, ARRIVED, DISCHARGED, GATE_OUT), location, event_time (local time with offset in clean), scraped_at (UTC). Clean is in event order; dirty is in scrape order.

exceptions.csv: EXC_ID (EXC-1xxxx Santos, 3xxxx Jakarta, 5xxxx Nhava Sheva, 7xxxx Shanghai), SHIPMENT_ID, DESK, PORT, TYPE (the desk's label), HS_CODE (as declared, which is wrong for WRONG_HS_CODE), OPENED_AT, STATUS, ASSIGNED_TO, RESOLVED_AT, RESOLUTION_CODE (what the desk did).

Types: INVOICE_MISMATCH (about half), MISSING_DOCS, CUSTOMS_HOLD, WRONG_HS_CODE, CARRIER_DELAY, DEMURRAGE_RISK, ROLLED_BOOKING; the US leg adds LATE_CUSTOMS_FILING and MISSED_TRUCK_APPT.

exception_events.csv: event_id (EV-, the exception's digits and a step number), exc_id, desk, event_type (OPENED, ASSIGNED, DOCS_REQUESTED, CUSTOMS_SUBMITTED, CUSTOMS_REJECTED, CUSTOMS_RESUBMITTED, RESOLVED), event_time (desk local time with offset), actor (an operator, system or customs). About 18,800 events for the core estate. Invoice mismatches, wrong HS codes and customs holds are resolved by filing with customs; a wrong fix is far more likely to come back. In clean, 10.8% of the invoice-mismatch cases sent to customs are rejected and resubmitted: the rework rate W10-05 baselines (the lesson's 11.2% is Halden's own measured figure).

documents.jsonl: doc_id, shipment_id, doc_type (commercial_invoice or packing_list), language, extracted_at, extractor, confidence, fields (invoice: invoice_no, invoice_date, shipper_name, consignee_name, incoterm, currency, amount, hs_code; packing list: packing_list_no, shipper_name, packages, gross_weight_kg, hs_code), and text, the document's own line in its language and number format. Names are printed as the shipper writes them; truth/entities.json maps every variant.

When an invoice disagrees with SAP, the reason is real and recorded in truth/exceptions_truth.csv: transposed digits, freight included (CIF on the invoice), a partial shipment, or a different currency.

emails.jsonl: email_id, received_at, mailbox, from (redacted), language, subject, body, has_attachment. Each email asks for a document, disputes an invoice, reports a delay, raises a customs query or changes a booking, with a deadline written the way people write it ("21/01 às 17h", "21/01 pukul 17.00", "21/01 शाम 5 बजे", "21 Jan, 17:00").

operators.csv: operator_id, name, site, role, languages, skills (invoice, documents, customs, reefer, trucking), shift_local. Rafaela Souza leads the São Paulo desk.

known_bad_shippers.csv: sap_vendor_id, shipper_name, reason (in the desk's language), added_by, added_on.

US leg (--sites all): isf_filings.csv (filed_at in the broker's Los Angeles time, vessel_loaded_at in Shanghai time), drayage_loads.csv (two-hour appointment windows, status DELIVERED, MISSED_APPT or SCHEDULED), truck_carriers.csv (USDOT- and MC-style numbers), check_calls.jsonl (carrier_ref is the MC number, status DISPATCHED, AT_TERMINAL, DELAYED or DELIVERED, free-text notes).

Truth

Defects in dirty/

Clean data is not perfect data. Label noise, invoices that disagree with SAP for a reason, and names printed in several ways are in both trees: they are the work, not defects.

Self-test

--self-test builds the estate four ways in memory and checks: two runs are byte-identical and match manifest.json; row counts are within 2% of their targets; for every defect, an independent detector finds exactly the lines in truth/defects.json and finds nothing in clean/; the desk-days missing from the event log are exactly the days a desk has no events; every alias resolves to a vendor; schema v2 changes only what is listed above; and --sites all adds only the US files and new-lane rows without changing a core row. It exits non-zero on any failure.

python3 _lab/estates/northvale/generate.py --out ./northvale-estate
python3 _lab/estates/northvale/generate.py --out ./northvale-estate --region us   # add the US-style subset
python3 _lab/estates/northvale/generate.py --self-test                            # about 4 s

A sample of 2,400 customers: 2,000 of the Mumbai bank and 400 of its Dublin subsidiary, Northvale Bank Europe. Times are UTC or local with an offset (Mumbai UTC+05:30; Dublin UTC+00:00, or +01:00 from 29 March 2026).

Files

Columns

cbs_customers.csv: CIF_ID (10 digits), FULL_NAME, DOB, MOBILE, EMAIL, CITY, STATE_OR_COUNTY, COUNTRY (IN or IE), HOME_BRANCH, SEGMENT (RETAIL, COMMERCIAL), KYC_STATUS (VERIFIED, PENDING, REJECTED), CUSTOMER_SINCE.

cbs_accounts.csv: ACCOUNT_ID, CIF_ID, PRODUCT (SAVINGS, CURRENT, CREDIT_CARD), CURRENCY, IFSC, IBAN, OPENED_ON, STATUS (ACTIVE, DORMANT, CLOSED). cbs_cards.csv: CARD_ID, ACCOUNT_ID, CARD_TYPE, NETWORK (VISA, RUPAY, MASTERCARD), PAN_MASKED, EXPIRY, STATUS.

transactions.csv: TXN_ID, ACCOUNT_ID, POSTED_DATE (the business date), POSTED_AT (local time with offset), AMOUNT, CURRENCY, DIRECTION (DR, CR), MERCHANT, CHANNEL (UPI, POS, ECOM, ATM, SEPA), REF.

beneficiaries.csv: BENEFICIARY_ID, CIF_ID, NAME, IFSC, ACCOUNT_NO, IBAN, BIC, ADDED_ON.

sf_person_accounts.csv: Id, Name, PersonEmail, PersonMobilePhone, CIF_Number__c, Branch__c, CreatedDate. okta_users.csv: id, login, firstName, lastName, mobilePhone, customerNumber, status, created.

sf_cases.csv: Id, CaseNumber, AccountId (the person account), Account_Number__c, CreatedDate, LastModifiedDate (UTC), Origin, Category__c (card_dispute, unauthorised_transaction, fees_and_charges, account_access, other), Priority, Status, Language__c (en, hi, or hi-en for Hinglish), Subject, Description, Disputed_Amount__c, CurrencyIsoCode. In clean, a card appears only as "card ending 1234". Both trees have 16-digit transaction references that look like card numbers and fail the Luhn check.

sf_case_export/: one case per line (attributes, Id, CaseNumber, Status, Category__c, AccountId, LastModifiedDate), ordered by LastModifiedDate, 200 to a page. export_log.json gives the method (clean: a query locator; dirty: LIMIT/OFFSET while agents edit cases), total_size, and each page's offset and record count. The dirty export holds as many records as the clean one, so a count check passes.

eval/golden_cases.jsonl: case_id, conversation_id, turn, split (dev, test), intent (W05-02's six), language, reason_code, message (masked: [CARD], [ACCOUNT]), labelled_by. 640 cases from 420 conversations; clean splits by conversation.

finops/: usage_hourly.csv has hour (UTC), api_key, bu, use_case, version, model (large, small, embed), calls, input_tokens, cached_tokens, output_tokens and cost_usd, priced from rate_card.json (illustrative prices). api_keys.csv registers each key to a business unit and cost centre; one 2025 proof-of-concept key belongs to nobody, in both trees. invoices.csv has each month's provider invoice (provider_direct) and the three shared pools; clean usage matches the provider line to the cent. Version 2.4.0 of dispute-summariser runs hot from 15 to 17 February: the retry-loop regression W09-09 describes.

US subset (--region us): card_disputes.csv (dispute_id, cardholder_id, state, opened_on, dispute_type, amount_usd, provisional_credit_on, resolved_on, outcome), credit_decisions.csv (decision_id, applicant_id, state, decided_on, product, decision approved, declined or counteroffer, reason_codes separated by semicolons, model_version) and reason_codes.csv (AA01 to AA08). The reason statements are written for the estate; W06-04's US lens covers what the rule asks of a reason. In the story Northvale doesn't lend in the US: treat the subset as the shape you'd meet at a US card issuer.

Truth

Defects in dirty/

--self-test builds the estate three ways and checks: two runs are byte-identical and match manifest.json; row counts match their targets; an independent detector finds exactly the lines (and the missing case numbers) in truth/defects.json for every defect, and nothing in clean/; every Salesforce and Okta record resolves to one customer; clean usage reconciles to the provider invoice and dirty usage doesn't; and --region us adds only the us/ files.

python3 _lab/estates/carevia/generate.py --out ./carevia-estate
python3 _lab/estates/carevia/generate.py --self-test     # about 3 s

Options: --out, --seed (2026) and --days (90, from Monday 5 January 2026, 7 to 365), as for the other estates. A sample of 3,000 members in Arizona, Florida, Georgia and Texas, 60 provider practices, and the prior-authorisation intake agent from week 3. Times are UTC unless a column says local; the contact centres are in Phoenix (UTC-7 all year) and Tampa (UTC-5, UTC-4 from 8 March 2026).

Files

Columns

members.csv: member_id, first_name, last_name, dob, sex, state, phone. coverages.csv: coverage_id, member_id, plan_id (PLN-CPPO-26, PLN-CHMO-26, PLN-MAHMO-26, PLN-MCD-26), start, end (empty while active), status. providers.csv: npi, name, specialty (orthopaedics, radiology, physical_therapy, neurology, primary_care), state, phone, fax.

claims.csv: claim_id, member_id, provider_npi, received_on, claim_type, status (paid, denied, pended), total_billed. claim_lines.csv: claim_id, line_no, service_date, cpt, icd10, units, billed.

pa_requests.csv: pa_id, member_id, provider_npi, cpt, icd10, received_at, channel (portal, fax, fhir, x12_278), urgency (standard, expedited), policy_id (IMG-LSPINE-04, IMG-KNEE-02, IMG-BRAIN-03, PT-VISITS-01), status (approved, pended_missing_info, denied, withdrawn, in_review), decided_at, decided_by (a nurse, RN-, or for a denial a physician, MD-: people decide, never the agent).

call_logs.csv: call_sid, member_id, started_at, centre (phoenix, tampa), agent_id, duration_s, reason. call_notes.jsonl: note_id, call_sid, member_id, agent_id, noted_at (the centre's local time with offset), text. Policy allows the member ID in a note and nothing else that identifies the member; clean notes follow it.

pa_agent/: intake_runs.jsonl has run_id, pa_id, started_at, path (primary, fallback, checklist), outcome (queued_for_nurse, manual_intake) and task_id. model_outputs.jsonl has run_id, pa_id, attempt, model, finish_reason and output, which should be one JSON object with pa_id, missing_items, urgency and summary. nurse_tasks.csv has task_id, pa_id, idempotency_key, created_at, queue, status.

traces/spans.jsonl: trace_id, span_id, parent_span_id, name, kind, service_name, host, start_time_unix_nano, end_time_unix_nano, status_code and attributes (GenAI semantic-convention names, as in W08-07). A trace is one run: invoke_agent prior-auth-intake with chat, retrieval, execute_tool check_member_eligibility (whose SERVER child runs on eligibility-api-1 or -2) and, when queued, send nurse-review and the consumer's process nurse-review minutes later. No member data in any attribute.

FHIR bundles (B7)

fhir_bundles/ holds 80 prior-authorisation requests that arrived as FHIR R4 bundles (type collection): Patient, the provider and payer Organizations, Coverage, ServiceRequest, DocumentReference for the documentation the service needs, and a Claim with use preauthorization. Codes use the CPT (http://www.ama-assn.org/go/cpt), ICD-10-CM (http://hl7.org/fhir/sid/icd-10-cm) and LOINC (http://loinc.org) system URIs. The folder is an inbox, not split into clean and dirty: some bundles are invalid, and some are valid but missing documentation, so a nurse would pend them. truth/fhir_bundles.json gives, per file, valid, the defects (code and path), complete and missing_documentation:

FHIR and electronic prior authorisation uses the folder: its exercise extends an intake until it agrees with this truth file on all 80 bundles, and 45 of them are complete on first submission.

Truth

Defects in dirty/

Claims for a member after a plan change follow the duplicate record in dirty: a consequence of C04 rather than a separate defect, so those lines aren't listed.

--self-test checks: two runs are byte-identical and match manifest.json; row counts match their targets; an independent detector finds exactly the lines (and the dropped span IDs) in truth/defects.json, and nothing in clean/; an independent FHIR validator reaches every verdict in truth/fhir_bundles.json; and members, providers, coverage, runs and tasks link up in clean.

python3 _lab/estates/kelverne/generate.py --out ./kelverne-estate
python3 _lab/estates/kelverne/generate.py --self-test     # about 5 s

Options: --out, --seed (2026), --days (90, from Monday 5 January 2026, 7 to 365) and --force, as for the other estates. The unemployment-insurance agency of the fictional State of Kelverne (state code AD, which is not a USPS code), the customer of engagements/kelverne/. The default window holds 19,595 claims, one per claimant: about 230 on an ordinary Wednesday, half as many again on Mondays, and nearly twice as many from 28 January to 8 February, when the Kestrel Falls Assembly Works plant closes (employer E400001). Phone and in-person claims are keyed into the 1990s mainframe; web claims come through the portal. Times are Kelverne local time with an offset (Central Time: UTC-6, UTC-5 from 8 March 2026).

Kelverne's benefit rules are invented for the estate, in the usual shape. The base period is the first four of the last five completed quarters before the claim. A claim is monetarily eligible with at least $2,000 of base-period wages in at least two quarters. The weekly benefit amount is the high quarter's wages divided by 26, in whole dollars, between $50 and $600; the maximum is the lesser of 26 weeks and a third of base-period wages. The first week is an unpaid waiting week. Separation, able-and-available and identity issues hold payments until an adjudicator decides them, and a determination can be appealed for 30 days after its notice.

On Wednesday 4 March 2026, from 09:00 to 15:00, the portal showed wrong next-payment dates: 600 extra calls about payment dates that morning carry the disposition portal_issue. That is the pack's incident exercise; the payments themselves are unaffected.

Files

Columns

claimants.csv: claimant_id, source_system, ssn (9NN-NN-NNNN), first_name, middle_name, last_name (as on the claimant's ID, accents included), dob, address_line1 (house number and street), address_line2 (the unit, or empty), city, state, zip, phone, email, preferred_language (en, es, vi), created_at, test_data. employers.csv: employer_id, fein, legal_name, naics_sector, town, state, employees, status, test_data.

wage_records.csv: record_id, employer_id, ssn, year, quarter, wages, reported_on (3 to 31 days after the quarter ends; a quarter not yet reported at the extract is absent).

claims.csv: claim_id, claimant_id, filed_date, channel (web, phone, in_person), program (UI, UCFE for federal civilian employees, UCX for former service members), last_employer_id, separation_reason (lack_of_work, quit, discharged, other), base_period (2024Q4-2025Q3), base_period_wages, high_quarter_wages, weekly_benefit_amount, max_benefit_amount, monetarily_eligible (Y, N), status (active, pending_adjudication, denied, monetarily_ineligible, inactive, exhausted), benefit_year_end.

issues.csv: issue_id, claim_id, issue_code (SEP-VQ, SEP-DM, AA-ILL, AA-CC, ID-VER, WS-INC, EARN-UR), issue_type, raised_by (portal_intake, mainframe, weekly_cert), detected_on, owner, due_date, status (open, decided). determinations.csv: determination_id, issue_id, claim_id, decided_on, decided_by, outcome (allowed, denied), reason_code, notice_mailed_on, appeal_deadline. appeals.csv: appeal_id, determination_id, claim_id, filed_on, filed_by, level, hearing_date, hearing_officer, outcome (affirmed, reversed, modified, withdrawn, pending), decided_on.

payments.csv: payment_id, claim_id, week_ending (a Saturday), certified_on, status (waiting_week, paid, processing, held, denied), amount, paid_on, method (direct_deposit, debit_card). About one first payment in six waits 10 to 22 days for a new bank account to be verified.

call_logs.jsonl: call_id, claimant_id, claim_id, started_at, channel (ivr, agent), transferred, language, intent (claim_status, payment_date, missing_document, appeal_question, weekly_certification, password_reset), wait_seconds (longer during the plant closure and the incident), handle_seconds, agent_id, disposition (resolved, callback, escalated, portal_issue), transcript (caller and agent or IVR turns). What a caller hears follows the claim's state on the day of the call.

documents.jsonl: doc_id, claim_id, claimant_id, doc_type (id_upload, separation_statement, determination_notice), created_at, source (claimant_upload, employer_response, system), language, fields (ID uploads: document, given_name, middle_name, family_name, dob; statements: author, issue_code, employer_id; notices: determination_id, outcome, reason_code, appeal_deadline), confidence (ID uploads) and text. Notices are written in the claimant's language.

Truth

The measures, for the default estate: 82.8% of first payments within 21 days of the end of the waiting week, 76.4% of nonmonetary determinations within 21 days of detection, and pending lower-authority appeals 16 days old on average. The US DOL's UI Performs targets are at least 87% of first payments within 14 days of the first compensable week (21 in waiting-week states), at least 80% of nonmonetary determinations within 21 days, and an average age of pending lower-authority appeals of 30 days or less (as of October 2026). Kelverne misses the first two: that is why the agency cares about the assistant.

Defects in dirty/

Instructions hidden in uploads are in both trees: they are the threat the pack's first exercise models, not a data defect.

--self-test checks: two runs, one in a separate process, are byte-identical and match manifest.json; row counts match their targets; an independent detector finds exactly the lines in truth/defects.json and nothing in clean/; every reference resolves, every determination is made by an adjudicator with a 30-day appeal window, and every claim's monetary determination follows from wage_records.csv; and the timeliness measures can be computed.

python3 _lab/estates/doorvale/generate.py --out ./doorvale-estate
python3 _lab/estates/doorvale/generate.py --self-test     # about 2 s

Options: --out, --seed (2026), --days (90, from Monday 5 January 2026, 7 to 365) and --force. Doorvale is the fictional US home-listings marketplace of engagements/doorvale/; Voxmere, a fictional voice-AI vendor, runs the voice agent. Ten invented towns in nine states (Calder Bay and Vista Serrano, CA; Northgate Hollow, WA; Mesa Alta, AZ; Silver Creek, CO; Cedar Run, TX; Lakemont, IL; Brightwater, FL; Fallon Ridge, GA; Pine Hollow, PA), four invented MLSs (Calvane, Ombrin, Tarnwell, Quessen) and a direct feed from brokerages, Doorvale Direct. The default window holds 8,000 listings (2,600 on the market on 5 January and 60 new a day), 2,772 web leads, 4,132 inbound calls and about 3,300 outbound callbacks. Listing and call times are UTC; leads and tours use the market's local time with an offset (Arizona stays on UTC-7).

The voice agent changed on 16 February 2026. Version va-1.3 answered "is it still available?" from a cached copy of the feed, and on about three in ten neighbourhood questions it answered in terms of who lives there. Version va-1.4 checks the listing's status live (a tool call that adds 140 to 420 ms to the turn) and declines to describe neighbourhoods by their residents. Both behaviours are in both trees: they are what the pack's exercises find, not data defects.

Files

Columns

listings.jsonl: ListingKey (DV and 7 digits; Doorvale Direct DVD), ListingId (the MLS number, CV26-01234), OriginatingSystemName, StandardStatus (RESO values: Coming Soon, Active, Active Under Contract, Pending, Closed, Withdrawn, Canceled, Expired), PropertyType (Residential, Residential Lease), ListPrice (a number; monthly rent for a lease), ClosePrice, CloseDate, StreetNumber, StreetName, StreetSuffix, UnitNumber, City, StateOrProvince, PostalCode, UnparsedAddress, BedroomsTotal, BathroomsTotalInteger, LivingArea, ListAgentKey, ListOfficeName, OnMarketDate, StatusChangeTimestamp, ModificationTimestamp. In clean, StandardStatus is the latest change in listing_status_changes.csv and ModificationTimestamp is when the feed received it.

listing_status_changes.csv: change_id, ListingKey, StandardStatus, source_timestamp, feed_timestamp (3 to 90 minutes later), OriginatingSystemName.

call_transcripts.jsonl: call_id (vx_ and 16 hex digits), direction, started_at, phone (E.164), caller_state (from the area code), market_state, lead_id (outbound), ListingKey, agent_version, recording_notice, ai_disclosure, outcome (availability_answered, tour_booked, handed_to_licensed_agent, not_now, opted_out, voicemail, no_answer, abandoned), handoff_agent_key, duration_ms and turns. A turn has speaker, start_ms, end_ms and text; caller turns add asr_confidence and barge_in; agent turns add interrupted and latency_ms (asr_final, llm_first_token, tts_first_audio, network and, for a live status check, tool), whose sum is the gap after the caller stopped. Caller text is lower-case ASR output. The first agent turn says it is an AI and that the call is recorded. Agent turn latency at p95 is about 1,290 ms on va-1.3 and 1,500 ms on va-1.4.

leads.csv: lead_id, created_at, source (listing_page_form, tour_request_form, saved_search_alert, partner_site), ListingKey, first_name, last_name, phone, email, state, assigned_agent_key, callback_requested, consent_id. consent_records.csv: consent_id, lead_id, phone, event (granted, revoked), method (web_form_checkbox, voice_request, sms_stop, preferences_page), system (consent_platform, voice_agent, sms_gateway), timestamp, text_version, text_shown, consumer_message. dnc.csv: phone, added_at, reason, source (consent_platform, legacy_import). In clean, every revocation reaches the list within 30 minutes and no outbound call follows one.

agents.csv: agent_key, full_name, brokerage, market, state, license_number (TX-T and 7 digits), phone, email, accepts_leads, test_data. tours.csv: tour_id, ListingKey, lead_id, call_id, agent_key, requested_at, scheduled_for, status (requested, confirmed, completed, no_show, canceled: a tour for a home that left the market is canceled), confirmed_at. eval/counsel_labels.csv: call_id, label (compliant, steering), note, labelled_by, labelled_on.

Truth

Defects in dirty/

--self-test checks: two runs, one in a separate process, are byte-identical and match manifest.json; row counts match their targets; an independent detector finds exactly the lines in truth/defects.json and nothing in clean/; every reference resolves, the feed agrees with the status history, every revocation reaches the do-not-call list and every call opens with the AI and recording disclosure; every agent turn starts exactly when its latency is spent and barge-in flags match the timings; and the stale answers and steering answers come only from va-1.3.