Architecting an Agentic Harness: Tool Use, Memory & Routing Patterns

The Blueprint: What Was Actually Built

Most "agent" demos are fragile scripts that break the moment an API returns a 500 error or a prompt exceeds the context window. The work dissected here—originally detailed by data4sci—is a deliberate departure from that fragility. It’s a structured agentic harness: a runtime environment that gives an LLM strictly governed access to external tools, a persistent memory system, and a routing layer that decides which tool to call based on semantic intent.

We aren't looking at a monolithic prompt that tries to do everything. We are looking at a control loop. The harness ingests a user objective, enters a reasoning step, selects a tool, executes it deterministically, feeds the result back into the context, and iterates until a termination condition is met.

The core stack is Python-native, leveraging LangChain for the orchestration scaffolding, ChromaDB for the vector store, and OpenAI's function-calling API as the reasoning spine. The novelty isn’t the individual components—it’s the strict separation of concerns between the Planner (the LLM), the Executor (the Python runtime), and the Memory (the vector store).

Why This Architecture Matters for Shipping Engineers

If you are a Forward Deployed Engineer (FDE) or a backend engineer tasked with putting LLMs into production, you aren’t selling a chatbot. You are selling deterministic access to a stochastic system. Customers don't care about your prompt template; they care that the invoice got processed correctly, or the support ticket was routed to the right queue.

This harness pattern matters because it solves the "vibe check" problem. In a naive implementation, you ask the LLM to do math or fetch a URL, and it might hallucinate the answer. In a harness, the LLM is never allowed to generate the final output for a tool. It only generates the parameters for the tool. The actual execution happens in a sandboxed Python function. This is the difference between a junior intern guessing and a senior engineer consulting a datasheet.

For FDEs embedding with customers to unlock operational value—a workflow we've dissected in our Palantir-style FDE approach—this architecture is the foundation for trust. You can't put an agent in front of a client if it can't provably execute a specific SQL query rather than dreaming one up.

Deep Dive: The Three Pillars of an Agentic Harness

Let’s break the monolith. The source implementation reveals three distinct subsystems that must be decoupled to avoid spaghetti-code collapse.

1. Tool Use: The Deterministic Execution Boundary

Tools are not prompts. They are Python functions with typed signatures. The harness uses the OpenAI function-calling schema to define a strict contract. When the LLM decides a tool is needed, it doesn't generate the tool's logic—it generates a JSON blob matching the function’s input schema.

Why this matters: The Python runtime validates this JSON against the schema before execution. If the LLM hallucinates a parameter, the harness throws a validation error and feeds that error back to the LLM for self-correction. This reflexion loop is the only way to achieve reliability above 95%.

# Simplified Schema Definition Pattern
tools = [
    {
        "type": "function",
        "function": {
            "name": "fetch_documentation",
            "description": "Retrieve technical docs for a specific product version",
            "parameters": {
                "type": "object",
                "properties": {
                    "product": {"type": "string", "enum": ["gateway", "ledger"]},
                    "version": {"type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$"}
                },
                "required": ["product", "version"]
            }
        }
    }
]

Notice the regex constraint on version. You aren't hoping the LLM behaves; you are forcing it to comply.

2. Memory: Short-Term Context vs. Long-Term Retrieval

Agents fail when they forget what they did three steps ago. The harness implements a dual-memory architecture:

• Ephemeral Memory: The standard message list (the chat history). This is the working memory. It's linear and limited by the context window.
• Persistent Memory (ChromaDB): A vector store that holds summaries of completed actions and retrieved documents. This is the agent’s "notepad."

When the context window gets crowded, the harness doesn't just truncate the oldest messages. It summarizes them and upserts the summary into ChromaDB. On the next reasoning step, the agent queries the vector store for relevant past context. This is mimicking human cognitive offloading—we don't keep every detail in our working memory; we write things down and refer back.

3. Routing: Semantic Intent Classification

Not every query needs the full agentic treatment. If a user asks "What is the capital of France?", you don't need to spin up a tool-execution loop. The harness implements a classifier gate.

Before entering the expensive agent loop, a lightweight classifier (often a smaller, cheaper model, or a simple embedding similarity check against a set of predefined intents) routes the query:

• Direct Answer: Bypass the agent, answer immediately.
• Tool Use: Enter the agent loop.
• Clarification: Ask the user for more info.

This routing pattern drastically reduces latency and cost. You aren't burning 10 seconds and 50k tokens to answer a simple question.

Implementation: How to Wire This Up Today

You don't need a PhD to build this. You need discipline. Here is the pragmatic path, cutting through the abstraction bloat.

Step 1: The Tool Registry Don't scatter your tools across your codebase. Build a central registry—a dictionary where keys are tool names and values are the callable functions and their schemas. This is your source of truth. If a tool isn't in the registry, the agent can't touch it.

class ToolRegistry:
    def __init__(self):
        self._tools = {}

    def register(self, func, schema):
        self._tools[func.__name__] = {"func": func, "schema": schema}

    def execute(self, name, params):
        if name not in self._tools:
            raise ValueError(f"Unauthorized tool: {name}")
        return self._tools[name]["func"](**params)

Step 2: The Control Loop The loop is simple. It’s a while loop with a max iteration guard (usually 10-15 steps). Inside the loop:

1. Inject the current message history + relevant vector store results into the prompt.
2. Call the LLM with the tool schemas.
3. If the response is a tool call: execute via the registry, append the result to history as a tool role message.
4. If the response is a final answer: break the loop and return.

Step 3: Memory Offloading Use a token counter (like tiktoken). When the cumulative tokens exceed a threshold (e.g., 70% of the model's limit), trigger a summarization step. Take the oldest messages, ask a cheap model (GPT-3.5 Turbo or Claude Haiku) to summarize them, store the summary in ChromaDB, and remove the raw messages from the active list.

For a practical example of wrangling data with these patterns, look at how we categorize bank CSV exports automatically with Gemini and Supabase. The same principle of deterministic parsing followed by LLM classification applies here.

The Engineer's Reality Check: Limitations and Trade-offs

Let’s kill the magic. This harness is not AGI. It is a state machine with a fuzzy logic controller.

The Fragility of Function Calling Even with strict JSON schemas, LLMs make mistakes. They might call a tool with the right structure but the wrong intent. For example, they might search for "error code 500" in the documentation when they should have searched for "internal server error." The harness has no semantic understanding of whether the correct tool was chosen—it only validates syntax. This means you still need robust error handling in the tools themselves.

The Cost Spiral Memory offloading is expensive. Summarization calls cost tokens. Embedding calls cost tokens. Vector storage costs infrastructure. If you aren't careful, your agentic loop costs $0.50 to answer a question that a static FAQ page could have answered for free. The routing classifier is not optional; it's the only thing keeping your cloud bill sane.

Latency is the UX Killer A loop of 5 tool calls, each requiring an LLM round-trip (500ms - 2000ms), plus vector retrieval, means your user is waiting 5-15 seconds. That's an eternity. You must implement streaming at every level—not just token streaming for the final answer, but status updates: "Searching documentation...", "Executing SQL...". Otherwise, the user thinks it's broken.

The "Why Not Just Code It?" Test Before you build an agent, ask: "Could I write a Python script with a few if/else statements that does this?" If the answer is yes, write the script. An agentic harness is the right tool only when the path to the answer is dynamically determined and requires reasoning over unstructured data.

FAQ: Tool Failures, Latency, and Context Limits

Q: What happens when a tool call fails after three retries? A: The harness must have a global exception handler. It should catch the failure, inject a clear error message into the LLM's context ("Tool X failed with error Y. Do not try this tool again."), and force the LLM to re-plan. If the agent is stuck, it should gracefully degrade to a "I'm unable to complete this task" message rather than looping infinitely.

Q: How do I prevent the agent from leaking PII into the vector store? A: You must sanitize data before it hits the embedding step. Implement a regex-based scrubber in your memory pipeline that redacts emails, phone numbers, and credit card patterns. Do not rely on the LLM to do this.

Q: Isn't LangChain too bloated for this? A: The source uses LangChain, but the core concepts are framework-agnostic. You can implement the control loop in 50 lines of pure Python using the OpenAI client directly. Use libraries for the hard parts (ChromaDB, tiktoken), but own the orchestration logic. Don't outsource your control flow to a framework you can't debug.

Q: How does this relate to building browser automation agents? A: The architecture is identical. You just swap the tool registry. Instead of documentation fetchers, you register Playwright actions (click, type, navigate). We explore this exact pattern in our guide on building a job-application autofill browser extension with Playwright and Gemini. The routing logic and memory management remain the same.

Q: Can I use open-source models for this? A: Yes, but with a caveat. The function-calling ability of open-source models has improved dramatically (as seen with recent models topping the agentic index, which we covered in our analysis of Qwen3.8 Max), but they still struggle with complex nested schemas. If you go open-source, you'll need to invest heavily in output parsers and retry logic to handle malformed JSON.