Retries, fallbacks and failing gracefully

The model can be wrong, malformed, refusing, slow, rate-limited or down, and every tool behind it can fail too. Handle each class deliberately. Retry only what a retry can fix (timeouts, connection errors, rate limits, overload) with capped exponential backoff and full jitter, never earlier than Retry-After, never past the deadline, never beyond a retry budget, and in exactly one layer. Make writes idempotent, because a timed-out write has an unknown outcome. Put a circuit breaker in front of each dependency so a dead endpoint costs milliseconds per request, not minutes. Fall back down a chain (smaller model, deterministic path, human) where each step is less autonomous than the last, and tell the user which path produced the result. Prove it with fault injection before the customer's outage does.

Key takeaways:

• Classify on the error body, not only the status: a 429 can be "slow down" or "you are out of credit".
• Full jitter spreads retries; Retry-After sets a floor; the deadline and the retry budget set a ceiling.
• One retry layer. SDK retries inside your retries inside workflow retries multiply load on a failing service.
• A fallback must lower autonomy, not just keep the lights on: a weaker path produces a flagged result for a human.
• Degradation is a feature with acceptance tests, not an absence of errors.