Routing models to fund the rollout

A model gateway is one internal endpoint every agent calls instead of calling providers directly. It holds provider keys, picks the model and provider per request, fails over on rate limits, errors and timeouts, enforces per-tenant budgets, keeps inference in permitted regions and emits the cost record finance needs. The savings come from routing: most enterprise traffic is easy, and a small model that passed the eval for that intent serves it far more cheaply. The discipline is that every fallback is a production model: same eval, same schema, same residency rule as the primary.

• Route by eval evidence: an intent moves to a cheaper tier only after it clears that intent's eval.
• Filter (residency, eval approval, schema) before you rank (price, latency, health).
• Reserve budget before each attempt; at the soft limit stop escalation, at the hard limit degrade to humans, never to silence.
• Partition caches by tenant and permission scope; keep semantic caching to non-personalised content.
• Record the model that actually served each attempt.