Can a MUD Evaluate LLMs? A $99 Game-Based Benchmark POC

What Actually Happened: The Crucible Experiment

A small team took a classic Multi-User Dungeon (MUD)—a text-based virtual world—and turned it into an LLM benchmark called Crucible. The entire proof of concept cost $99 in API credits. They threw frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) into a parser-driven text adventure and measured whether they could navigate, plan, and survive.

The setup was brutally simple: each model received the same textual description of a room, the same available exits, and the same inventory. The goal was to solve multi-step puzzles that required spatial reasoning, object permanence, and long-horizon planning—the kind of stuff that doesn't show up in a multiple-choice MMLU question.

The headline finding? Models that ace standard reasoning benchmarks struggled with basic navigation. They'd drop quest-critical items, forget locked doors they'd already found, and wander in loops. The $99 experiment exposed a gap that $10M evaluation suites somehow missed.

Why a MUD? The Engineering Insight Behind Game-Based Evals

Standard LLM evals are static. A prompt goes in, a token sequence comes out, a regex or classifier scores it. That pipeline assumes the world is a single-turn, context-free transaction. Real engineering work—especially Field Distillation Engineering (FDE)—never looks like that.

A MUD forces the model into a stateful, partially observable environment. Every action changes the game state. The model doesn't get to re-roll. If it drops the rusty key in the dungeon foyer, that key stays there until it walks back. This exposes failure modes that single-turn evals completely miss:

• State tracking decay: The model forgets what it's carrying or where it's been after 15-20 turns.
• Goal drift: It starts with a clear objective ("find the amulet") and gets distracted by shiny room descriptions.
• Causal confusion: It tries actions that make linguistic sense but violate game physics ("open door" when the door was already described as "stuck fast").

For an FDE deploying LLMs in customer environments, these are the exact failure modes that turn a demo-day miracle into a production incident. A chatbot that forgets the user's account tier three messages in. An agent that loses track of which API call it already made. Standard benchmarks won't catch that. A MUD will.

The $99 Architecture: How It Was Built

Here's the flow. No magic, just clean plumbing:

The stack is deliberately boring:

1. MUD Engine: They used Evennia, a Python-based MUD framework. It handles the world model, room connections, object persistence, and command parsing out of the box. No need to build a game loop from scratch.
2. Thin Orchestration Layer: A Python script that converts Evennia's game state into a structured text blob (room description, visible exits, inventory, recent action history) and feeds it into a templated prompt.
3. LLM API Calls: The model gets the prompt, returns a natural language action ("go north", "take lantern", "unlock door with brass key"). That string gets parsed back into an Evennia command.
4. Scoring: Quest objectives are defined in a JSON schema ("reach room ID 42", "possess item 'silver_amulet'"). The scoring module checks state after every turn and at termination.

The $99 budget broke down roughly as: $20 for Evennia hosting (a tiny cloud VM), $70 for API calls across all models (thousands of turns total), and $9 for miscellaneous logging storage. The entire thing ran in a weekend.

The Prompt Template (Simplified)

You are an agent in a text adventure game. Your goal: {quest_description}.

Current location: {room_name}
Description: {room_description}
Exits: {exits}
Inventory: {inventory}

Recent actions:
{action_history}

Respond with exactly ONE game command. Valid commands: go <direction>, take <object>, drop <object>, inventory, look, unlock <object> with <object>, use <object> on <object>.

The constraint to output a single command is critical. Without it, models tend to narrate their reasoning, plan out loud, and then forget to actually act. This is the same problem you see when an LLM agent "thinks" its way into a timeout on a customer ticket.

What the Results Told Us (and What They Didn't)

Claude 3.5 Sonnet won. It completed the most quests with the fewest wasted turns. GPT-4o was competitive but prone to hallucinating objects that didn't exist ("take sword" in a room with no sword). Gemini 1.5 Pro had strong spatial reasoning but struggled with the strict command format, frequently outputting prose instead of a valid action.

But the raw scores aren't the interesting part. The interesting part is what broke:

These are not "the model is dumb" problems. They're architecture problems. The model has no external working memory, no explicit world model, and no mechanism to reflect on its own confusion. Sound familiar? It's exactly what happens when you wrap an LLM in a while loop and call it an agent.

What the experiment doesn't prove: This is a proof of concept, not a rigorous benchmark suite. One MUD, a handful of quests, no statistical power analysis. Don't cite Crucible scores in a procurement decision. But do pay attention to the methodology—it's a template for building evals that actually stress-test the capabilities you care about.

Build Your Own: A Practical Blueprint

You can replicate this for your own domain. The pattern generalizes far beyond text adventures. Here's the recipe:

Step 1: Define Your Environment State Machine

A MUD is just a state machine with rooms as nodes and exits as edges. Your customer's SaaS product is also a state machine—accounts, configurations, feature flags, support tickets. Map it. What are the valid states? What are the valid transitions? If you can't draw this, your agent can't navigate it either.

Step 2: Build a Thin Simulator

You don't need Evennia. A 200-line Python class with a state dict and a step(action) method works. The key requirement: actions must have side effects that persist. If your simulator is stateless, you're just building another multiple-choice test.

class SupportTicketSimulator:
    def __init__(self):
        self.state = {
            "ticket_status": "open",
            "customer_tier": "enterprise",
            "assigned_agent": None,
            "resolution_attempts": 0,
            "knowledge_base_articles_retrieved": []
        }
    
    def step(self, action: str) -> dict:
        # Parse action, update state, return observation
        # This is where you encode your domain rules
        pass

Step 3: Wire the LLM In

The prompt template above is a starting point. For production-grade eval, add:

• Structured output: Force JSON responses with a schema, not free-text commands.
• Termination conditions: Max turns, success criteria, and a timeout.
• Action validation: Reject invalid moves and feed the error back. This is how you test error recovery.

Step 4: Score on Trajectories, Not Just Outcomes

Did the model reach the goal? Great. But also measure:

• Path efficiency: Ratio of optimal steps to actual steps.
• Error recovery rate: How many invalid actions before a valid one?
• State retention: Does it remember critical context after N turns?

This trajectory-level scoring is what separates a toy eval from something that predicts production behavior. If you're building a resume tailoring agent or an on-call incident summarizer, the same pattern applies—test the full workflow, not just the final output.

The FDE Angle: Why This Matters in the Field

Field Distillation Engineers live at the boundary between a model's training distribution and a customer's messy reality. You're the one who discovers that the agent works beautifully on demo data and falls apart when the customer's API returns a 429 with a malformed error body.

Game-based evals matter for FDEs because they test agentic resilience—the ability to pursue a goal through an environment that pushes back. When you're embedding with a customer, you need to know which failure modes your LLM stack is vulnerable to before you're on a call explaining why the automation just deleted a production config.

This also connects to the handoff between FDE and core engineering. When you surface a systematic failure ("the agent can't track state beyond 12 turns"), that's not a support ticket. That's a product requirement. Game-based evals give you the evidence to make that case with data, not anecdotes.

And if you're thinking about building internal tools—a smart clipboard or a Discord FAQ bot—the Crucible pattern scales down nicely. A lightweight stateful simulator can tell you more about your prompt engineering than a hundred static test cases.

FAQ

Q: Is Crucible a replacement for standard benchmarks like MMLU or HumanEval? A: No. It measures a different thing—agentic reasoning in a stateful environment. Use both. Standard benchmarks tell you about knowledge and single-turn reasoning. Game-based evals tell you about planning, memory, and error recovery.

Q: Can I use this to evaluate fine-tuned or open-weight models? A: Absolutely. The architecture is model-agnostic. Swap the API call for a local inference endpoint running Ollama or vLLM. The $99 budget was for hosted APIs; local inference is effectively free if you have the hardware.

Q: What's the catch with MUD-based evals? A: They're high-variance. A model might fail a quest because of one bad random action, not because of a fundamental flaw. Run multiple seeds and report distributions, not point estimates. Also, building a good quest is a design problem—it's easy to accidentally create puzzles that only make sense to humans.

Q: How do I convince my team this is worth the effort? A: Run the $99 experiment yourself on your own domain. A weekend of work, a handful of API credits, and you'll have concrete examples of your agent failing in ways your current eval suite never caught. That demo is worth more than any whitepaper.

Q: Does FDE Coach offer training on building these kinds of evals? A: The hands-on projects in FDE Coach's curriculum—building agents, summarizers, and retrieval systems—all incorporate stateful evaluation patterns. When you build a resume tailoring agent or an incident summarizer, you're not just wiring up an API; you're learning to design tests that catch the failure modes that matter in the field. That's the muscle this $99 experiment exercises, and it's the same muscle you build across every project in the program.