LLMs Can't Jump: The Spatial Reasoning Ceiling for Vision Models

The Setup: A Simple Grid, a Stubborn AI

Imagine a 7x7 grid. You place a red token at position (3,4) and a blue token at (5,2). You ask a state-of-the-art vision-language model (VLM) like GPT-4V or Gemini Pro: "To move from red to blue, do you go up, down, left, or right?"

The answer isn't just wrong; it's confidently nonsensical. The model might say "down and right" when the correct vector is "down and left." It fails to trace a simple path.

This isn't a trick question. A four-year-old can solve it. Yet, as the position paper LLMs Can't Jump demonstrates, this is a catastrophic failure mode for frontier models. The paper introduces a benchmark of simple grid-based tasks—counting houses on a street, navigating mazes, identifying relative positions—and the results are brutal. VLMs perform near chance on tasks requiring genuine spatial understanding, even when they correctly identify every object in the scene.

This is the "hard ceiling" for vision models. They see the pixels but don't perceive the space.

The Core Failure Mode: Disembodied Vision

Why does an entity that can pass the bar exam fail to navigate a 2D grid? The paper posits a fundamental architectural constraint: VLMs process images as sequences of flattened patches, not as coherent spatial topologies.

When you feed an image to a VLM, a vision encoder (like ViT) slices it into small, non-overlapping patches. These patches are linearized and fed into a transformer alongside text tokens. The model learns correlations between specific patches and text labels—"this patch of fur correlates with the word 'cat'." But it doesn't maintain a persistent, metric map of where those patches reside in relation to one another.

It's the difference between knowing you are in a room with a table and a chair, and knowing the chair is two feet behind you. The former is semantic recognition; the latter is spatial reasoning.

The "Can't Jump" paper calls this the "disembodied" nature of current VLMs. They lack a world model that respects Euclidean geometry. They can't jump because they don't have legs; they have a massive lookup table of visual features.

The VLM Architecture Bottleneck: Text Trapped in a 2D Prison

Let's look under the hood. This is the standard VLM inference flow:

The critical failure point is the transition from the Vision Encoder to the LLM Backbone. The spatial grid of the image is compressed into a sequence of tokens. The LLM sees a 1D list of vectors, not a 2D map. Positional embeddings give the LLM a sense of patch order, but this is a weak substitute for true coordinate geometry. The model cannot perform mental rotations, distance calculations, or path tracing because the data structure it uses—a text token sequence—is fundamentally unsuited for it.

It's akin to describing a chess board by listing the pieces left-to-right, top-to-bottom, and then asking the listener to calculate a knight's move. The information is technically there, but the representation makes the computation intractable.

Why This Breaks Your Real-World Pipeline

For an FDE shipping customer-facing AI, this isn't an academic curiosity. It's a production landmine. Consider these scenarios:

1. Document Parsing and RAG

You're building a RAG system that ingests complex PDFs—invoices, contracts, architectural diagrams. The VLM correctly extracts text via OCR. But it fails to understand that the "Total" field at the bottom-right is the sum of the line items above it, not just a random number. If you used a Screenshot-to-Code Agent to parse a UI mockup, it might correctly identify buttons but place them in the wrong layout because it doesn't grasp the spatial constraints.

2. Robotics and Embodied AI

You instruct a robot: "Pick up the leftmost apple." The VLM identifies three apples. It cannot reliably determine "leftmost" from the camera's perspective. It hallucinates an ordering. The robot grabs the wrong one. This isn't a motor control problem; it's a spatial reasoning failure in the perception layer.

3. UI Automation and Testing

You use an agent to navigate a dashboard. You ask it to "click the button to the right of the search bar." The model knows what a search bar is and what a button is. It doesn't know how to compute "to the right of." Your automation script flails.

4. Medical Imaging

A VLM spots a tumor. A radiologist needs to know if it's anterior to a critical blood vessel. The model provides a description but flips the spatial relationship, creating a severe safety risk.

In each case, the model's object recognition is perfect, but its spatial judgment is broken. This is the exact failure mode the "Can't Jump" paper warns about.

Engineering Mitigations: How to Make VLMs 'Jump'

We can't fix the underlying architecture today. But as engineers, we can build scaffolding that compensates. Here are practical, high-signal mitigations:

1. Explicit Coordinate Extraction (The "Grid Overlay" Pattern)

Don't ask the VLM for spatial reasoning. Ask it for coordinates. Overlay a grid on the image or use a specialized object detection model (like YOLO or Grounding DINO) that returns bounding boxes with pixel coordinates.

Then, use deterministic code (Python) to compute spatial relationships.

# Pseudocode: Don't ask the LLM "which is to the left?"
def is_left_of(box_a, box_b):
    return box_a['x2'] < box_b['x1']

# Use the VLM only for the semantic task of identifying objects
objects = vlm_query(image, "Return bounding boxes for all apples")
leftmost = min(objects, key=lambda obj: obj['x1'])

This pattern is crucial for building reliable Codebase Q&A Tools that need to reason about UI screenshots or architecture diagrams.

2. Chain-of-Thought with Tool Use

Force the model to externalize the spatial problem. Instead of asking "Is the red block on the left?", prompt: "1. Print the coordinates of the red block. 2. Print the coordinates of the blue block. 3. Write Python code to compare their x-coordinates."

By offloading the Euclidean computation to a Python interpreter, you bypass the LLM's spatial blind spot. This is the "Tool-Use" paradigm. The LLM acts as a semantic router, not a geometer.

3. Multi-Modal, Multi-Pass Prompting

Use a two-stage pipeline. First, use a vision model to generate a dense textual description of the scene, explicitly listing spatial relationships in a structured format (JSON). Second, feed that JSON to a text-only LLM (like GPT-4 or Mixtral) for logical reasoning.

// Stage 1 Output (VLM)
{
  "objects": [
    {"label": "red square", "relative_position": "top-left quadrant"},
    {"label": "blue circle", "relative_position": "bottom-right quadrant"}
  ]
}

This converts a spatial problem into a textual logic problem, which LLMs handle far better. This architecture is similar to the Multi-Agent Research Assistant pattern, where you decompose a hard problem into specialized agents.

4. Synthetic Data Augmentation for Fine-Tuning

If you must fine-tune a smaller VLM for a specific spatial task (e.g., warehouse robot navigation), generate vast amounts of synthetic data. Create 3D renders of your environment with randomized object placements and automatically generated labels for spatial relationships. This doesn't make the model "jump," but it memorizes a specific spatial layout well enough for a constrained domain.

A Balanced Take: Is This Really a Hard Ceiling?

It's tempting to read the "Can't Jump" paper and declare VLMs fundamentally useless for spatial tasks. That's an overreaction.

The counter-argument: Humans don't do pure Euclidean geometry in their heads either. We rely on heuristics, experience, and eye movements. The fact that VLMs fail on abstract grids doesn't mean they can't navigate real-world environments, where texture, lighting, and context provide massive shortcuts. Many robotics systems successfully use VLMs for navigation by pairing them with 3D depth sensors and SLAM algorithms.

The truth: The paper identifies a genuine architectural limitation, not a temporary bug. The "patchification" of images is a lossy compression of spatial topology. Until we have architectures that maintain an explicit 3D world state—perhaps through Neural Radiance Fields (NeRFs) or Gaussian Splatting integrated into the reasoning loop—VLMs will remain brittle on tasks requiring precise metric understanding.

For the Forward Deployed Engineer, the takeaway is clear: Trust VLMs for semantic extraction, never for spatial computation. Pair them with deterministic geometric libraries. Treat them as an OCR engine with a vocabulary, not as a CAD program. This is the same pragmatic, builder-first mindset you need to prepare for the FDE interview loop or to ship customer-facing features on a tight cadence.

FAQ: Spatial Reasoning for Engineers

Q: Does this failure apply to video models too? A: Yes, and it's often worse. Video models add a temporal dimension, but the spatial reasoning per frame suffers from the same patchification problem. Motion is often inferred from optical flow heuristics, not true 3D understanding.

Q: Can't we just increase the image resolution? A: Higher resolution helps with recognizing small objects but doesn't fix spatial reasoning. The model still sees a sequence of patches. It just has more patches. The fundamental problem is the lack of a coordinate system.

Q: Are there any models that can do spatial reasoning? A: Specialized models trained on synthetic 3D data (like some robotics perception stacks) perform better in narrow domains. General-purpose VLMs that natively integrate depth maps or point clouds are an active research area but not yet widely available.

Q: How do I detect this failure mode in my own product? A: Build a small evaluation set of "relative position" questions specific to your domain. For a document parser, ask "Is the signature block below the total amount?" For a UI agent, ask "Is the submit button to the right of the cancel button?" If accuracy is below 90%, you need the engineering mitigations described above.

Q: Is this why self-driving cars don't use LLMs? A: Precisely. Autonomous vehicles rely on LiDAR, radar, and deterministic sensor fusion algorithms to create a metric 3D map of the world. They use deep learning for object detection, but the path planning is done with classic robotics algorithms that respect Euclidean geometry. An LLM is not in the control loop.