Fan-out and sandboxed parallel work

When an agent must process hundreds or thousands of independent records (exceptions at a port cut-off, bills of lading in a backlog, invoices at month end), doing them one after another is the latency problem. Fan-out runs many at once and fan-in collects the results. The speed-up is capped by the tightest shared resource (the carrier portal's session limit, the customer's SAP rate limit, your provider's tokens per minute), so you bound concurrency per resource, time out each attempt, retry transient failures with jittered backoff, coalesce duplicate requests, and report partial failures instead of failing the batch. Parallelism cuts wall-clock time, not tokens: the bill is the same or higher. For work that can wait hours, provider batch APIs trade latency for a 50% discount. When agents write and run code as part of the work, that code is untrusted: it runs in an isolated, ephemeral sandbox with no secrets, no network by default and hard resource limits.

Key takeaways:

• Speed-up ≈ min(concurrency limit, number of items) for uniform work, eroded by tail latency and retries.
• One semaphore per shared resource, sized from that resource's real limit, not one global number.
• Every item returns a typed result with a status. The batch reports successes, failures by reason, and whether it met its threshold.
• Batch APIs (Anthropic, OpenAI) cost half and complete within 24 hours; they are for backfills and evals, not for cut-off night.
• Model-generated code runs in a sandbox: fresh filesystem, no credentials, egress denied by default, CPU, memory and time limits.