Why We Deprecated Our LLM Router: Cost, Complexity, and Diminishing Returns

The Pitch: Why Everyone Ships a Router

The promise of an LLM router is seductive. You have a fleet of models—cheap, fast ones like GPT-4o-mini or Claude Haiku, and expensive, deliberate ones like Claude Opus or o1. A router sits in front of them, examines the incoming prompt, and decides: does this really need the heavy artillery? If the user just wants a grammar fix, route to the small model and pocket the 90% cost savings. If they're debugging a race condition in distributed systems, pay up for the smart model.

On paper, it's a perfect optimization problem. Reduce average cost per token while maintaining quality. The industry has gone all-in. OpenRouter, Martian, Unify, Portkey, and a dozen open-source frameworks all sell variations on this idea. You train a classifier, embed prompts, or use an LLM to judge complexity, then dynamically dispatch.

We drank the Kool-Aid. At Manifest, we embedded a routing layer inside our backend. It ran for months. We measured every metric. Then we deleted it.

This is the autopsy.

What We Built: The Manifest Architecture

Our router followed the canonical pattern. We had three tiers of models: a fast tier (Haiku-level), a mid tier (Sonnet-level), and a heavy tier (Opus-level). A lightweight classifier—essentially a distilled BERT variant fine-tuned on our own prompt logs—scored every incoming request on a "complexity" axis. The classifier looked at prompt length, presence of code blocks, semantic density, and domain-specific signals like whether the user was referencing internal API documentation.

We added a response cache keyed on prompt embeddings to short-circuit identical or near-identical requests. An observability layer logged routing decisions, latency, and user feedback (thumbs up/down) so we could close the loop and retrain the classifier weekly.

The initial results looked great. In staging, we routed 40% of traffic to the fast tier, 45% to mid, and 15% to heavy. Projected cost savings: 35-40% compared to sending everything to the mid tier. The engineering team celebrated. We shipped to production.

The Cracks Appear: Complexity Debt

Within two weeks, the on-call rotation noticed a pattern. PagerDuty alerts fired for "routing misclassification"—a metric we'd instrumented based on user-reported dissatisfaction correlated with the routed tier. Users who got the fast model on prompts that looked simple but required deep reasoning (e.g., "Why is my Kubernetes pod crashing?" with a 200-line YAML dump) would get a shallow answer, then immediately re-ask. The re-ask hit the mid or heavy tier, so they eventually got a good answer, but the user experience degraded.

We tuned the classifier. Then we tuned it again. Each tuning cycle required:

1. Labeling sessions: Engineers manually reviewed hundreds of prompt-response pairs to decide which tier should have handled them.
2. Retraining the classifier: A fine-tuning job on our internal GPU cluster, plus evaluation against a holdout set.
3. Shadow deploys: Running the new classifier in shadow mode alongside production for a week to compare decisions without affecting users.
4. Gradual rollout: Canary deploys, monitoring, rollback when precision dropped.

This is machine learning ops, not a routing feature. The team spent more time maintaining the router than building product features. The classifier itself became a living system with its own tech debt: training data drift, concept drift as models got updated upstream, and edge cases where our complexity score was perfectly anti-correlated with actual user satisfaction.

Then the models changed under us.

The Economic Reality Check

Anthropic shipped Claude 3.5 Haiku. It was faster and smarter than the previous Haiku, nearly matching Sonnet on many tasks. OpenAI dropped GPT-4o-mini pricing by 60%. The cost gradient we'd optimized for flattened dramatically.

Here's the brutal math we ran internally:

The router saved us $0.30 per million tokens—roughly 10%. But the classifier itself consumed compute, eating into those savings. The p99 latency was worse because a misroute to the fast tier followed by a fallback to heavy added two full round-trips. And the engineering cost dwarfed the infrastructure savings: 40 hours a month of senior engineering time costs far more than the $2,000-3,000 we saved on API bills.

We also discovered a behavioral pattern: users who hit the fast tier and got a mediocre answer didn't just re-ask. They lost trust. Some churned. The "soft" cost of degraded UX doesn't show up on a cloud bill, but it shows up in retention curves.

The Engineering Pivot: Static Rules Won

We replaced the entire router with a 40-line YAML config and a deterministic dispatcher. The logic:

routing_rules:
  - name: summarization
    pattern: "summarize|tldr|summary"
    model: fast
  - name: code_generation
    pattern: "write (a|some) (code|function|script)"
    model: heavy
  - name: code_review
    pattern: "review|check|audit"
    context_has_code: true
    model: heavy
  - name: default
    model: mid
  fallback:
    on_error: retry_with_heavy
    max_retries: 1

This is regex-based routing with a single fallback tier. It covers 80% of the value with zero machine learning. The pattern matching is transparent, debuggable with grep, and doesn't drift. When Anthropic ships a new model, we update one line in a config file.

The key insight: the complexity distribution of real user prompts is bimodal, not continuous. Users either ask trivial things ("summarize this meeting") or genuinely hard things ("debug this distributed deadlock"). The mushy middle—prompts where a router might genuinely optimize—is small. A static rule set captures the modes. The classifier was solving a problem that didn't exist at meaningful scale.

Our observability layer didn't go away. We still log model, latency, and user feedback. But instead of feeding a training pipeline, that data feeds a weekly 15-minute review where we adjust the YAML rules if needed. That review has resulted in exactly one rule change in three months.

What This Means for Forward Deployed Engineers

If you're an FDE building AI features for customers, this story should hit close to home. The job of an FDE isn't to ship the most elegant architecture—it's to ship something that works, stays working, and doesn't generate support tickets at 2 AM.

Router complexity is a classic FDE trap. You're integrating an LLM into a customer's workflow, the customer asks about cost optimization, and the router pattern looks like the "right" engineering answer. But the customer doesn't care about your routing architecture. They care about:

• Reliability: Does it work every time?
• Latency: Is it fast enough for their SLA?
• Cost: Is the bill predictable?

A static routing config with a single fallback delivers all three better than a trained classifier in most cases. The case study on deploying LLM features at a regulated enterprise reinforces this: simplicity wins when you're shipping to a customer who will run your code for years with minimal oversight.

This also ties into the broader FDE skillset around debugging and prototyping. When a customer reports that "the AI is giving dumb answers," you need to trace the routing decision instantly. With a regex rule, you grep the logs. With a classifier, you're debugging a black box. The week-in-the-life FDE workflow is built on fast debugging loops—routers break that loop.

When You Should (and Shouldn't) Build a Router

Don't build a router if:

• Your model cost differential is less than 5x between tiers
• You have fewer than 10M tokens/day in throughput (the engineering cost dominates)
• Your prompt complexity distribution is bimodal (most real-world products)
• You don't have a dedicated ML ops team
• Your users are sensitive to latency tail (p99)

Consider a router if:

• You're spending >$50K/month on LLM APIs and the cost differential between your cheapest and most expensive model is >10x
• You have a continuous complexity distribution (e.g., an educational product where questions span grade 1 to PhD)
• You have a team dedicated to maintaining the routing infrastructure
• You're willing to invest in a proper evaluation framework with human-annotated quality scores

Even then, start with static rules and measure. You might find, as we did, that the 80/20 solution is the 100% solution.

For engineers building RAG systems or internal tools, the pattern holds. Our codebase Q&A bot guide deliberately uses a single model with no routing—because the complexity distribution of "search my repo" queries is tightly clustered. Adding a router would be pure overhead. Similarly, the Slack digest bot runs on Cloudflare's free tier with a single model; routing would break the free-tier economics entirely.

FAQ

Q: Doesn't a router let you use the newest, most expensive models without blowing your budget?

Only if the expensive model is genuinely needed for a subset of traffic and that subset is predictable. In our case, the expensive model was needed less than 15% of the time, but the cost of misclassifying even 2% of prompts into the cheap tier created enough user friction to offset the savings. A static rule that routes code generation to the expensive model and everything else to the mid-tier captured 90% of the value with zero misclassification risk.

Q: What about using an LLM as the router instead of a trained classifier?

This pattern—"use GPT-4o-mini to decide whether to call GPT-4o"—is popular. It adds latency (you're making two API calls instead of one) and cost (the router call itself costs tokens). It also inherits all the non-determinism of LLMs. Our testing showed that LLM-based routers had a 5-8% misclassification rate on our workload, which erased the cost savings when you account for fallback calls.

Q: Did you consider caching as an alternative to routing?

We did, and we still use semantic caching heavily. Identical or near-identical prompts get served from cache, bypassing the model entirely. Caching gave us 20-25% cost reduction with zero quality impact. It's a strictly better optimization than routing for most products. Start there.

Q: Isn't this just a reflection of Manifest's specific traffic patterns?

Yes, and that's precisely the point. Router performance is intensely workload-dependent. The only way to know if a router works for you is to measure your actual prompt complexity distribution, your actual cost differential, and your actual user tolerance for quality variance. Most teams skip this measurement step and jump straight to building. Don't.

Q: What's the one metric that should kill a router project?

P99 latency. If your routed traffic has a worse tail latency than your single-model baseline, the router is failing. Users remember the slow responses, not the median. Our p99 went from 1.8s to 4.2s under routing. That alone justified the deprecation.

---

The original Manifest write-up that inspired this analysis is available at manifest.build/blog/why-we-deprecated-our-llm-router.