OpenRouter That Learns: Optimizing Model Routing with Usage Data

What Just Dropped: A Self-Optimizing LLM Gateway

A team at Experiential Labs open-sourced a project that rethinks how we route prompts to large language models. The core idea is simple but powerful: instead of hardcoding rules or relying on a static leaderboard, the router records every prompt, response, and feedback signal, then uses that history to make smarter routing decisions in the future. They call it an "open OpenRouter that learns."

Here’s the plain breakdown. You send a prompt to the gateway. The gateway logs it. You get a response back from one of several supported providers—OpenAI, Anthropic, Gemini, or local models. Later, you (or your app’s users) provide explicit or implicit feedback: thumbs up/down, a corrected output, or a metric like task success. That feedback is stored alongside the original prompt and model choice. Over time, the system builds a dataset of what worked for which kinds of requests. When a similar prompt arrives, the router consults this history and picks the model most likely to succeed, balancing cost, latency, and quality based on real outcomes rather than benchmarks.

This isn't just another proxy with a config file. It's a closed loop. The routing table isn't static JSON; it's a living artifact shaped by production traffic. For engineers who have been duct-taping together model fallback chains or manually A/B testing providers, this is a fundamental shift from guesswork to evidence.

The Architecture: Log, Learn, Route

The system runs as a FastAPI server with a pluggable backend. At its heart, three components interact:

The gateway intercepts every request. The logger embeds the prompt (using a lightweight embedding model) and stores it in a vector database alongside metadata: which model was used, what the response was, and any feedback that arrived later. The router, when faced with a new prompt, queries the vector store for similar historical requests. It aggregates the performance data for those neighbors—success rates, latency, cost—and applies a configurable scoring function to pick the best model for this specific prompt.

The feedback collector is the secret sauce. It can accept explicit signals (a rating endpoint) or implicit ones (did the user regenerate? Did they accept the suggestion in an IDE?). Each feedback event updates the score of the original routing decision in the vector store. The next time a semantically similar prompt arrives, the router has one more data point. This is online learning at the application layer, and it requires zero manual annotation.

Why a Learning Router Matters for Engineers

Static routing is brittle. You pick a model based on a benchmark like MMLU or HumanEval, set it in a config, and pray it holds up across every user query. It won't. A model that aces Python coding might hallucinate on a legal contract summary. A cheap model might be perfect for chit-chat but useless for structured data extraction. Static routing forces you to choose one model for all traffic or build complex, hand-tuned classifier chains to split requests.

A learning router solves this by making routing a data problem. If your application handles customer support tickets, the router will learn that GPT-4o is overkill for "reset my password" but essential for "explain my billing discrepancy." It learns from your actual distribution of prompts, not from a sanitized eval set. This is the difference between buying tires based on a Formula 1 track test and buying tires based on the potholed streets you actually drive.

For engineers building AI features, this means:

• Cost optimization without quality regressions: Route simple queries to cheaper models automatically.
• Provider redundancy: If one API goes down, the router can fall back based on learned quality, not just a static priority list.
• A built-in eval framework: The logged data becomes a goldmine for fine-tuning. You now have a dataset of real prompts and the model that handled them best, tagged by outcome.

This approach also mirrors a broader engineering principle: instrument first, optimize later. Too many teams jump straight to model selection without a feedback mechanism. They’re flying blind. This gateway bakes observability into the routing layer itself.

The Forward Deployed Lens: Closing the Loop with Customers

If you’ve read about what a Forward Deployed Engineer actually does in a week, you know the job is about embedding in a customer’s environment and solving problems that off-the-shelf tools can’t. A learning router is a classic FDE weapon. Here’s why.

FDEs often deploy AI prototypes into messy, real-world data streams. The customer’s users don’t behave like a benchmark. They type typos, mix languages, and ask questions no one anticipated. An FDE who ships a static model router will spend the next two weeks firefighting edge cases. An FDE who ships a learning router can point to the feedback loop and say: “It’ll get better every day. Here’s the dashboard.”

This pattern—ship a working system with a tight feedback loop, then let the data drive improvements—is pure FDE playbook. You’re not trying to predict every edge case before launch. You’re building the mechanism that discovers and fixes them in production. The Experiential Labs gateway gives you that mechanism for model selection.

Consider a customer support automation use case. You deploy an AI agent that answers questions using a knowledge base. On day one, you route everything to Claude 3.5 Sonnet because it’s safe. By day seven, the router has learned that 40% of queries (simple FAQ lookups) are handled perfectly by a much cheaper model. You’ve just cut the customer’s inference bill without touching a line of code. That’s the kind of win that turns a prototype into a long-term engagement. If you’re interested in building similar RAG-backed agents, the pattern in Build a Discord Community FAQ Bot Backed by Your Docs shows how to structure the retrieval side, and this router would sit in front of the LLM call.

Deploying and Trying It Today

The repository is MIT-licensed and straightforward to get running. You’ll need Python 3.10+, API keys for the providers you want to use, and a vector store. The project supports Chroma (local, zero-config) and Pinecone for production.

Quick start:

git clone https://github.com/experientiallabs/experiential.git
cd experiential
pip install -r requirements.txt

Configure your providers in a .env file:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...

Launch the gateway:

uvicorn app.main:app --reload

Send a prompt just like you would to any OpenAI-compatible endpoint:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Summarize this legal document..."}]
  }'

Notice "model": "auto". That’s the trigger for the learning router. You can also specify a model explicitly to bypass routing and force a particular provider—useful for debugging or when you have a strong prior.

After you get a response, you can submit feedback:

curl -X POST http://localhost:8000/v1/feedback \
  -H "Content-Type: application/json" \
  -d '{
    "request_id": "...",
    "score": 1,
    "comment": "Perfect summary"
  }'

The router uses this feedback immediately. There’s no batch retraining step. The vector store is updated in real time, and the next similar query benefits from the signal.

For a production deployment, you’ll want to configure the scoring function. The default balances cost and success rate, but you can weight latency, set hard cost caps, or define custom metrics. The configuration lives in a YAML file where you specify provider priorities, cost weights, and the embedding model to use for similarity search.

A Balanced Look: Strengths and Trade-offs

This approach is not magic. It’s a pragmatic engineering solution with clear upsides and some sharp edges you need to understand before betting on it.

Strengths:

• Adapts to your data distribution. Benchmarks measure average performance; your users are not average.
• Reduces the cost of experimentation. You don’t need to design a separate eval harness. Production is your eval, and feedback is the label.
• Vendor-agnostic by design. The router treats providers as interchangeable resources. You can add a new model, and it will earn traffic proportionally to its real-world performance.
• Open source. You can inspect the routing logic, modify the scoring function, and self-host everything. No black-box SaaS routing tax.

Trade-offs:

• Cold start problem. On day zero, the router has no history. It must fall back to a default policy (round-robin, cheapest-first, or a static priority). You need to decide that fallback and accept that early routing will be suboptimal. The system gets better with volume.
• Feedback quality is everything. If your feedback signal is noisy—users thumbs-down correct answers because they don’t like the truth—the router learns the wrong lesson. Implicit signals (did the user copy the code? did they immediately ask a follow-up?) are often cleaner than explicit ratings. You’ll need to design your feedback collection carefully.
• Embedding drift. The router uses semantic similarity to group prompts. If your prompt distribution shifts significantly (e.g., you launch a new feature), old history may become irrelevant. The system needs enough ongoing traffic to keep the vector store fresh.
• Not a replacement for evals. This is online optimization, not offline validation. You still need a regression test suite for critical behaviors. The router tells you which model is best on average for a prompt cluster; it doesn’t guarantee a specific output format or safety property.

For teams that already have observability pipelines—logging prompts and responses to a data warehouse—this router is a natural evolution. You’re already collecting the data; this just closes the loop and makes it actionable in real time. If you’re building agents that chain multiple LLM calls, the pattern extends naturally. Each call in the chain can be routed independently, and the feedback can propagate backward through the chain if you track causal links. The architecture shares DNA with the agentic workflows described in Build a WhatsApp Support Agent Backed by Your Docs Using n8n and Supabase, where multiple tool calls and LLM invocations need to be orchestrated and optimized.

FAQ: Costs, Cold Starts, and Privacy

Does this increase my inference costs?

In the short term, maybe marginally, because you’re adding an embedding step and a vector lookup per request. Those costs are tiny—fractions of a cent—compared to LLM inference. In the medium term, the router should reduce your total spend by shifting traffic to cheaper models when quality data says it’s safe. The break-even point depends on your traffic volume and the cost gap between your models.

How do I handle the cold start?

Set a sensible default model (your current best pick) and a fallback chain. The router’s configuration lets you specify a default_model that handles all requests until enough feedback accumulates. You can also seed the vector store with synthetic data—a few hundred representative prompts with hand-labeled model preferences—to give the router a head start. This is manual work upfront but pays off quickly.

What about data privacy?

Everything runs in your infrastructure. Prompts and responses are stored in your vector database. If you’re handling sensitive data, you control the retention policies and access controls. The embedding model runs locally (the project uses all-MiniLM-L6-v2 by default, which is small and fast). No data leaves your environment unless you route a prompt to an external API—and that’s the same data you’d be sending anyway. The router doesn’t add a new third-party dependency for data storage.

Can I use this with local models?

Yes. The gateway supports any OpenAI-compatible endpoint, which includes local servers like Ollama, vLLM, or llama.cpp. You can route between cloud providers and local models seamlessly. This is powerful for hybrid deployments where you want sensitive prompts to stay on-prem but are willing to use cloud models for less sensitive, high-complexity tasks. The router learns which prompts need the big cloud model and which are handled fine by your local 7B.

How is this different from an LLM proxy like LiteLLM?

LiteLLM and similar proxies focus on providing a unified API interface with static routing rules (least-cost, round-robin, fallback chains). They’re excellent at normalization. Experiential adds the learning layer: routing decisions that improve over time based on feedback. You could potentially build this on top of LiteLLM, but Experiential bakes it into the core loop. The two approaches are complementary, not mutually exclusive.

Is this production-ready?

It’s an early-stage open-source project. The core loop works, but you should expect to invest time in tuning the scoring function, setting up monitoring, and hardening the deployment for your scale. That said, the architecture is sound, and the codebase is clean enough that an experienced engineer can fork it and adapt it to production needs in a week. For FDEs working on customer prototypes that need to ship fast, that’s a reasonable trade-off.