Schema Harness Hit ~99% on Arc-AGI-3 Without Giant LLMs – Here's How

The Raw Result: Schema Harness vs. Arc-AGI-3

The Arc-AGI-3 public benchmark just got a new leader. Schema Harness, a system that explicitly avoids relying on frontier-scale LLMs, scored approximately 99% accuracy on the public evaluation set.

To put that in context: the ARC (Abstraction and Reasoning Corpus) challenge was designed by François Chollet to be resistant to memorization. It tests fluid intelligence—the ability to infer a rule from just a few input-output grid examples and apply it to a novel test input. Giant, trillion-parameter models have historically struggled here, often plateauing in the 30-50% range on prior versions.

Schema Harness didn't just edge past them. It nearly aced the test. And it did so by brute-forcing reasoning structure, not parameter count.

The source paper and implementation details are available at the project page: https://schema-harness.github.io/.

The Architecture: Why 'Schema' Beats 'Scale'

Most LLM-based approaches to ARC treat the problem as a pixel-prediction task. You feed the model a sequence of colored grids and ask it to hallucinate the next grid. This works poorly because the model lacks a native understanding of the discrete, rule-based transformations (rotations, object permanence, tiling, symmetry) that ARC relies on.

Schema Harness flips the script. It operates in three distinct phases:

1. Discrete Perception, Not Continuous Pixels

Instead of treating the grid as raw RGB values, the system segments it into distinct objects based on connectivity and color. This is a classic computer vision approach—connected components analysis—not a learned embedding. This immediately collapses the problem space. The model isn't guessing pixel colors; it's reasoning about "blue 2x2 square at position (3,4)."

2. Domain-Specific Language (DSL) for Grid Transformations

The core innovation is a custom DSL of grid-manipulation primitives. Think of it as an instruction set architecture for visual reasoning. It includes operations like:

• crop(obj)
• rotate(obj, deg)
• translate(obj, dx, dy)
• fill(obj, color)
• overlay(base, top)

Schema Harness doesn't learn these operations; they are hand-crafted, deterministic building blocks.

3. Program Synthesis via Search

Given an input-output pair, the system searches over compositions of these DSL primitives to find a program (a sequence of operations) that perfectly transforms the input into the output. This is brute-force symbolic reasoning. It leverages a library of known schemas—common transformation patterns—to prune the search tree.

When the verifier confirms a candidate program works on all given examples, it applies that program to the test input. No neural network generates the final grid. The DSL interpreter renders it deterministically.

Why Engineers and FDEs Should Care

This isn't just an academic curiosity. The Schema Harness result is a practical masterclass in system design for working engineers, especially Forward Deployed Engineers (FDEs) who live at the intersection of customer problems and technical constraints.

The Anti-Fragility of Deterministic Logic

LLMs are probabilistic. They hallucinate. In a production pipeline—say, an agent that classifies bank transactions or routes support tickets—a 1-in-20 error rate is a customer escalation waiting to happen. Schema Harness shows that for well-scoped, rule-based problems, a symbolic engine wrapped in a thin neural layer is far more reliable than a pure end-to-end deep learning approach.

If you're building a personal finance categorizer over bank CSV exports, you don't need a model to "understand" money. You need a system that applies regex patterns, date heuristics, and merchant-name matching with perfect recall. Schema Harness is the extreme, high-performance version of that philosophy.

The Cost of Inference

Running a 70B-parameter model to sort colored squares is absurdly expensive. Schema Harness can likely run on a single CPU core. For an FDE managing customer cloud costs or deploying edge AI, this is the difference between a viable product and a cost-overrun disaster. The technique mirrors the efficiency gains we chase when automating RouterOS configs with LLMs—use the LLM for intent parsing, but let deterministic scripts handle the actual config generation.

The FDE Mindset: Demos That Never Break

A recurring nightmare for FDEs is a live demo failing because the model hallucinated. Schema Harness, by design, cannot hallucinate the final output grid. It either finds a program that exactly matches the training examples, or it fails explicitly. This binary success/failure mode is a superpower in customer-facing environments. You can build trust by showing a system that gracefully says "I don't know" instead of silently producing garbage. For more on the reality of shipping demos under pressure, see a week in the life of an FDE.

How to Run Schema Harness Locally Today

The project is open-source. Here’s how to get it running on your machine. We'll assume a Unix-like environment with Python 3.10+.

Step 1: Clone and Setup

git clone https://github.com/schema-harness/schema-harness.git
cd schema-harness
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Step 2: Run on a Single ARC Task

ARC tasks are JSON files containing train/test pairs. Grab a sample from the public ARC-AGI-3 dataset.

python run.py --task data/tasks/sample_task.json --max_search_time 60

The --max_search_time flag bounds the program synthesis. For most tasks, the correct program is found in under 5 seconds.

Step 3: Interpreting the Output

The system outputs the generated DSL program. For example:

# Generated program for task: fill_blue_rectangle
program = [
    ("find", {"color": "blue"}),
    ("bounding_box", None),
    ("fill_rect", {"color": "red"})
]

This is human-readable and auditable. You can step through it with a debugger, unlike a neural network's latent space.

Step 4: Extending the DSL

To add a new primitive (e.g., flip_diagonal), edit dsl/primitives.py:

def flip_diagonal(grid, obj_id):
    # Custom numpy/torch logic here
    return transformed_grid

Register it in the schema library, and the search engine will automatically consider it for future tasks.

A Balanced Engineer's Take

Schema Harness is not AGI. Let’s get that out of the way. It solves ARC-AGI-3, a specific, closed-world benchmark. It doesn't generalize to natural language, robotics, or open-ended reasoning. The hand-crafted DSL is a form of hard-coded prior knowledge, which limits its scope to grid-based puzzles.

However, dismissing it as "just a brute-force solver" misses the point. The engineering lesson is about architectural leverage. The system combines the flexibility of a neural network (for perception and hypothesis generation) with the safety and precision of a symbolic verifier. This hybrid architecture is immediately applicable to real-world problems:

• Document parsing: Use an LLM to propose extraction schemas, but verify them against the document's layout boxes deterministically.
• Code generation: Use a model to suggest a function, but validate it with a type checker and test suite before execution.
• Customer support: Classify intent with an embedding, but route to a deterministic decision tree for actions.

For FDEs, this is the playbook for building AI triage agents that draft replies without hallucinating promises you can't keep. The model proposes; the schema verifies.

The limitation is the human effort required to build the DSL. Schema Harness required experts to encode geometric reasoning primitives. For a new domain—say, molecular biology—you'd need a new DSL. This is where the FDE skill set shines: sitting between the customer's domain and the engineering team, translating messy real-world rules into a structured, verifiable system. It's the same muscle you build when measuring time-to-value and adoption metrics.

FAQ

Does Schema Harness use any LLM at all?

The core program synthesis engine is purely symbolic and does not require an LLM. However, the complete system uses a lightweight vision model for the initial object segmentation step. This model is small (a few million parameters) and could be replaced with classical CV algorithms like flood-fill connected components.

Can I use this for non-grid problems?

Directly, no. The DSL is specific to 2D grid transformations. The architectural pattern—perception, DSL-based synthesis, verification—is universally applicable. You'd need to define a new DSL for your domain.

How does this compare to o3 or other frontier models on ARC?

Frontier models like GPT-4 or o3 use massive neural networks and chain-of-thought prompting. They achieve moderate scores (often 30-60%) at extremely high computational cost. Schema Harness achieves ~99% on the public set at a fraction of the cost by leaning on symbolic reasoning. Note that the private test set results may differ, and the system was likely tuned on the public set.

Is this a step towards AGI?

No. It's a step towards understanding the type of intelligence ARC measures: fluid reasoning over discrete, rule-based systems. It proves that for these problems, explicit symbolic search outperforms implicit neural reasoning.

How can I learn to build systems like this?

The core skill is translating domain expertise into structured representations. At FDE Coach, we focus on exactly this: the engineering craft of bridging customer needs with robust, verifiable AI systems. It's the same discipline required to deploy a RAG chatbot over your PDFs that doesn't hallucinate, or build a resume tailoring agent that rewrites your CV without inventing experience.