LLM-Extensible Software: Why Static APIs Are Dead

We are witnessing a violent collision between two software paradigms. One is the deterministic, contract-first world of REST and gRPC, where an endpoint expects structured JSON and rejects anything else with a stern 400 Bad Request. The other is the probabilistic, fuzzy-logic world of Large Language Models (LLMs), where natural language is the input, and the output is a best-guess token prediction.

Jeremy Morrell’s piece on Extensible Software in the age of LLMs nails the core tension: we are moving from a world where humans write glue code to integrate systems, to a world where the LLM is the glue code. This isn't about adding a chatbot to your app. It’s about designing software that treats the model as a runtime engine capable of dynamically composing logic.

For engineers, and particularly Forward Deployed Engineers (FDEs) who live in the messy gap between a product’s rigid boundaries and a customer’s chaotic reality, this shift is fundamental. You are no longer just piping data; you are designing a constrained playground for a non-deterministic actor.

The Implicit API: When the Schema Isn't the Contract

Traditionally, extensibility meant defining a strict plugin interface. Think of a VSCode extension: it has a specific package.json manifest, specific activation events, and a TypeScript API surface that is rigorously typed. If you write a byte outside the schema, it fails.

LLMs invert this. The “API” becomes an implicit surface defined by natural language descriptions of tools. You don’t pass a JSON blob to a specific endpoint; you pass a list of function signatures and their docstrings to a model, and say, “Figure it out.” The contract isn't enforced by a compiler or a JSON schema validator; it’s enforced by the model’s semantic understanding of the prompt.

This is terrifying for a backend engineer used to static analysis. But it’s liberating for an FDE. When a customer asks for a feature that combines data from a legacy SOAP endpoint and a modern GraphQL layer, you don’t have to write and deploy a new microservice. You define the two tools, hand them to the LLM, and let it orchestrate the data wrangling in a single generate() call.

Why Static APIs Crumble Under LLM Pressure

Static APIs assume the client knows exactly what it wants and how to ask for it. An LLM client doesn’t. It has a vague goal and needs to explore your system to satisfy it. If your API is a rigid set of CRUD endpoints, the LLM is forced into a brittle, multi-step “try and pray” loop—it calls GET /users, gets a 404 because it should have been GET /entities?type=user, and gets stuck.

To be extensible by an LLM, your software needs to be navigable. It needs to be self-describing. This is why the industry is rapidly converging on the tool-use pattern. You don’t expose a raw HTTP endpoint; you expose a function with a name, a description, and a JSON schema for its parameters. The LLM “reads” the description and decides to invoke it.

{
  "type": "function",
  "function": {
    "name": "search_inventory",
    "description": "Search the warehouse inventory by product name or SKU. Returns available stock counts.",
    "parameters": {
      "type": "object",
      "properties": {
        "query": {
          "type": "string",
          "description": "The search term, e.g., 'blue widget' or 'SKU-123'"
        },
        "warehouse_id": {
          "type": "string",
          "description": "Optional warehouse ID to scope the search"
        }
      },
      "required": ["query"]
    }
  }
}

This is the new extensibility primitive. It’s not a REST endpoint; it’s a semantic hook.

The Runtime Shift: From Calling Functions to Writing Them

Morrell’s most provocative point is that LLMs don’t just call tools—they write code on the fly. We’ve seen this with OpenAI’s Code Interpreter, where the model generates Python to solve a data analysis task, executes it in a sandbox, and returns the result.

This is the ultimate form of extensible software: a system where the LLM doesn’t just select from a fixed menu of pre-built functions, but synthesizes new logic to handle edge cases. For an FDE deploying an LLM feature at an enterprise customer (as detailed in our case study on deploying an LLM feature in 6 days), this is the difference between saying “Sorry, the system can’t do that” and “Let me write a quick script for you.”

The architecture shifts from a fixed call graph to a loop. The LLM proposes a solution (code or tool calls), the sandbox executes it, the output is validated, and if it fails, the error is fed back into the LLM for a retry. This “self-healing” execution loop is the killer app for extensibility.

Architecting for Extensibility: The Tool-Use Spectrum

Not all extensibility is created equal. You need to decide where on the spectrum your system lives:

1. Fixed Tool Set (Deterministic Routing): You have 10 tools. The LLM picks one. This is the safest and most predictable pattern. It’s perfect for replacing a traditional chatbot decision tree.
2. Dynamic Tool Composition (Chaining): The LLM can call multiple tools in sequence, passing the output of one as the input to the next. This is where you need to start worrying about state management and context window pollution.
3. Code Synthesis (The Deep End): The LLM writes and executes arbitrary code. This is maximum flexibility, but requires a hardened sandbox. If you’re embedding with a customer who has strict data residency requirements, you’re likely stuck at Level 2 for a while.

For FDEs, the skill isn't just in prompt engineering; it's in tool design. A great tool definition is a UX exercise for a non-human user. The description must be so clear and the parameters so constrained that the LLM has a narrow path to success. Vague tools produce hallucinated parameters.

Practical Patterns: Validation Loops and Guardrails

The biggest lie in LLM demos is that the output is immediately usable. In production, the raw output of an LLM is toxic waste until it passes a validation gate. When designing an extensible system, you must treat the LLM as an untrusted, junior developer.

Never pipe LLM-generated SQL directly to your production database. Never execute LLM-generated code without a review step, unless it’s in a sandbox with no network access. The pattern is always: Generate -> Validate -> Execute -> Validate.

• Pydantic for Output Parsing: Use Instructor or Outlines to force the LLM to output valid JSON that conforms to a strict schema. If it fails, retry with the validation error.
• Semantic Guardrails: Use a second, smaller, faster LLM (like a fine-tuned classifier) to check if the generated text violates a safety policy or contains PII before it reaches the user.
• Human-in-the-Loop (HITL): For high-stakes actions (sending an email, transferring funds), the LLM should draft the action and queue it for human approval. This is the ultimate extensibility pattern: the LLM does 80% of the work, the human provides the judgment.

This mirrors the workflow we explored in What a Forward Deployed Engineer Actually Does in a Week—the engineering work is often about building the safety rails, not the happy path.

The Forward Deployed Perspective: Integration Over Abstraction

Abstraction layers like LangChain try to hide the complexity of LLM interaction behind a unified interface. But extensibility breaks down when the abstraction leaks—and it always leaks. An FDE knows that the customer’s specific JSON schema, with its weird nested arrays and inconsistent date formats, will break a generic parsing library.

When designing extensible LLM systems in the field, prefer thin, transparent layers. Write the raw API call to the model yourself. Manage the conversation history as a simple list of dictionaries. This gives you the granularity to handle the specific failure modes of the customer’s environment.

Extensibility in the FDE context means letting the customer define new tools via a simple YAML config file, not waiting for a product sprint. You build the engine; they define the tools. This is the essence of unlocking trapped value, a topic we dissect in How Palantir-Style FDEs Embed with Customers.

The Catch: Determinism, Cost, and the "Vibe Check"

Let’s be clear about the downsides. Extensible software via LLMs is non-deterministic. The same input can produce different outputs. This violates a core assumption of most testing frameworks. You can’t write simple unit tests; you need “vibe checks”—evaluations that use another LLM to judge if the output is semantically correct.

Cost is also a silent killer. An extensible system that dynamically chains 15 tool calls and generates 10,000 tokens of code per request will burn through your inference budget. You need to implement caching aggressively. If a user asks a similar question, retrieve the previous successful tool-calling sequence from a vector store instead of re-planning from scratch.

Finally, debugging is a nightmare. When a static API fails, you get a stack trace. When an LLM-driven system fails, the model confidently executes the wrong tool and returns a plausible-looking but incorrect answer. Your observability must capture the full trace of model reasoning, tool selection, and parameter generation.

FAQ: Extensible LLM Architectures

Q: Is the tool-use pattern just RPC with extra steps? A: Conceptually, yes. But the “extra steps” are the point. RPC requires the client to know the exact procedure name and parameter types at compile time. Tool-use allows the client to discover the procedure at runtime based on a semantic description. It trades compile-time safety for run-time flexibility.

Q: Won't fine-tuning replace the need for dynamic tool composition? A: No. Fine-tuning bakes knowledge into weights, which is great for teaching a model a specific style or static knowledge base. But extensibility requires connecting to live systems (databases, CRMs) whose state changes. You can’t fine-tune a model on your current inventory count. Tools are the bridge between static weights and dynamic state.

Q: How do I prevent the LLM from calling a dangerous tool? A: Authorization is not the model’s job. Just because the LLM can call a tool doesn’t mean the user is authorized. Your tool execution layer must authenticate the user independently of the LLM’s decision. The LLM requests the tool; your middleware checks the user’s JWT scopes before executing it. Never trust the model’s judgment on security.

Q: I'm an FDE. What's the fastest way to prototype this for a customer? A: Start with a single, powerful tool that wraps a read-only SQL query against a replica database. Give the LLM a strict prompt: “Generate a SQL query to answer the user’s question. Only use SELECT statements.” This solves 90% of enterprise data questions without the risk of writes. We break down the tactical execution of this exact pattern in our guide on writing customer-facing technical docs that actually get read, because often the tool is useless if the user doesn't know how to prompt it.

Q: How does this relate to the "self-improving" models I keep hearing about? A: It’s the application layer of that concept. Models like those discussed in our piece on Ornith-1.5 and self-scaffolding focus on the model improving its own reasoning. Extensible software focuses on the model improving its utility by reaching out into the digital world. The two concepts converge when a model not only calls a tool but rewrites the tool definition to be better next time.