Can Coding Agents Handle a Million-Line Codebase? Databricks Put Them to the Test

The Experiment: No Toy Problems Here

Databricks engineering didn't run another benchmark on a curated set of LeetCode problems or isolated function stubs. They pointed coding agents at their actual monorepo—the one that builds their products, contains millions of lines of code across multiple languages, and has the kind of tangled dependency graphs that make seasoned engineers reach for coffee.

The test was straightforward: give agents a natural language task description, let them search the codebase, read files, write code, run tests, and iterate. No hand-holding. No pre-selected context windows. Just the raw repo and a problem statement. The goal was to measure how close we are to the "drop an agent into a massive codebase and have it ship a feature" dream that AI tooling vendors love to pitch.

They used a custom evaluation framework that scored agents on multiple axes: task completion, code correctness, test pass rate, and—critically—whether the agent's changes integrated cleanly with the existing architecture. This last part is where most toy benchmarks fall apart. Generating a function that passes unit tests is one thing; understanding that you shouldn't add a circular dependency to the build graph is another entirely.

The agents tested included both open-source and commercial offerings. The team ran hundreds of tasks across different difficulty levels, from simple bug fixes to cross-module refactors that required understanding invariants spread across dozens of files.

Why This Matters for Engineers

If you're a working engineer, you've probably had this thought: "Sure, Copilot is great for autocompleting a function, but can it actually understand our codebase?" This benchmark gets at exactly that question.

The distinction between "/v1/completions" and "/v1/agents" isn't just an API version bump. Completion tools work on the file you're looking at. Agents are supposed to navigate the codebase, build a mental model, and execute multi-step plans. The Databricks test exposes the gap between those two paradigms.

For engineering leaders evaluating coding agents for their teams, the results provide a concrete data point: how much productivity gain is realistic today, and what kinds of tasks you should (and shouldn't) delegate. For individual contributors, it's a preview of what your workflow might look like in 6-12 months—and what skills will remain uniquely human.

The benchmark also matters because it tests on a codebase that's actively maintained by hundreds of engineers. This isn't a snapshot of a repo from 2019 with clean boundaries. It's the messy, evolving reality of production software, complete with legacy patterns, inconsistent documentation, and those three files everyone is afraid to touch.

Where Agents Excelled (and Where They Faceplanted)

The Good

Agents handled localized bug fixes surprisingly well. Given a stack trace and a pointer to the general area, agents could trace through the code, identify the root cause, and produce a fix that passed tests. For bugs contained within a single module or two, success rates were high enough to be genuinely useful.

Boilerplate generation was another clear win. Adding a new API endpoint that follows existing patterns? Agents nailed it. They could find similar endpoints, understand the conventions, and produce consistent code. This is the kind of work that eats up real engineering hours and requires little creative insight.

Documentation updates tied to code changes were also strong. Agents that could read the diff and update relevant docs saved the kind of busywork that engineers chronically skip.

The Ugly

Cross-cutting concerns exposed the agents' shallow understanding. Tasks that required modifying invariants across three or more modules—say, changing a data structure that touched serialization, business logic, and the API layer—caused agents to produce code that compiled but violated architectural contracts. The agent would fix the immediate error but miss that the change broke a downstream assumption documented only in a design doc from 2022.

Build system changes were a disaster. When tasks required modifying Bazel BUILD files or Maven POMs to add dependencies, agents frequently created circular dependencies or pulled in unnecessary transitive deps. The agents lacked the global graph view that human engineers develop over months of working in a codebase.

Test generation was hit-or-miss. Agents could write tests for the happy path, but edge cases—especially those involving concurrency, resource exhaustion, or interactions between components—were consistently missed. The tests looked plausible but didn't actually prevent regressions.

Large-scale refactors that touched 50+ files exposed context window limitations. Agents would lose the thread midway through, producing inconsistent changes that broke the build in subtle ways.

Benchmark Breakdown: The Numbers

Databricks published a detailed breakdown across task categories. Here's the summary:

A few things jump out. First, the cliff at 10+ files is real and steep. Second, even the "good" categories hover around 60-70%, which means human review is still non-negotiable. Third, the time savings on successful tasks are significant—a 3-minute agent run versus 30+ minutes of human debugging adds up across a team.

The raw success rates don't tell the full story. Databricks also measured "useful partial completion": cases where the agent didn't fully solve the task but produced code that a human could complete faster than starting from scratch. That metric was 30-40% higher across all categories, suggesting agents are currently better as accelerators than autonomous developers.

Practical Playbook: Using Agents on a Real Codebase Today

If you want to experiment with coding agents on your own large codebase, here's the engineer's guide based on what worked (and didn't) in the Databricks benchmark.

Start with the Right Tasks

Don't throw an agent at your gnarly authentication refactor. Start with:

• Bug fixes where you can provide a stack trace or reproduction steps
• Adding endpoints/models that follow existing conventions
• Writing documentation for recently changed code
• Generating test suites for well-defined modules

These tasks have high success rates and low blast radius. They build trust while actually saving time.

Context Engineering is Everything

Agents aren't mind-readers. The difference between a 15% and 60% success rate often came down to how well the task was framed. Effective prompts included:

• File paths or module names to scope the search
• Explicit "don't touch X" constraints
• Links to existing code that demonstrates the desired pattern
• A clear definition of done ("the existing tests in Y must still pass")

Think of it like onboarding a new team member—you wouldn't say "fix authentication" and walk away. You'd point them to the relevant modules, explain the architecture, and set boundaries.

Build Agent-Friendly Code

Some codebases are more agent-friendly than others. Databricks found that agents performed significantly better when:

• Modules had clear, consistent naming conventions
• Code was well-typed (agents leverage type information heavily)
• Tests were discoverable and ran quickly
• Architecture decisions were documented in-code, not just in external wikis

This aligns with best practices for maintainable code generally—agents just amplify the pain of messy codebases.

The Review Workflow

Treat agent output like a PR from a junior engineer you trust but verify. The Databricks team found that:

• Agent PRs needed the same level of review as human PRs
• The review was faster because the code followed patterns, but required more scrutiny on architectural decisions
• Running the full test suite (including integration tests) caught failures that unit tests missed
• Agents sometimes introduced subtle bugs that passed tests but violated invariants—these required human architectural knowledge to catch

Tool Selection

The benchmark tested multiple agent frameworks. The key differentiator wasn't the underlying LLM (all used capable models) but the agent's ability to:

• Search the codebase effectively (grep, AST search, semantic search)
• Maintain state across multiple tool calls
• Recover from errors (compilation failures, test failures) without losing context

When evaluating tools, test them on your actual codebase, not a demo repo. The performance gap between clean demo code and production code is massive.

A Balanced Take: Neither Hype nor Doom

The headline result is that coding agents are useful today for specific, well-scoped tasks on large codebases. They're not replacing senior engineers, and they're not vaporware. They occupy a pragmatic middle ground: a tool that can meaningfully accelerate development when used with the right expectations and workflows.

The 72% success rate on single-file bugs is genuinely impressive. That's real time saved on tasks that make up a significant fraction of engineering work. But the 23% on large refactors is a reality check for anyone expecting to fire their platform team.

What's most interesting is the trajectory. The agents that performed best used techniques that are improving rapidly: better retrieval, longer context windows, and more sophisticated planning. The gap between "useful assistant" and "autonomous developer" is closing, but it's measured in years, not months.

For engineers, the practical takeaway is: start experimenting now. The tools are good enough to learn on, and the skills of prompt engineering, agent workflow design, and AI code review will be increasingly valuable. But don't bet your sprint commitments on an agent shipping a cross-module refactor by Friday.

The Databricks benchmark is available in full on their engineering blog with detailed methodology and per-task breakdowns. It's worth reading if you're considering deploying agents on your own codebase.

FAQ

Q: Which coding agent performed best? A: Databricks didn't crown a single winner. Performance varied significantly by task type. The key insight was that agent architecture (search strategy, error recovery, state management) mattered more than the underlying model for most tasks.

Q: Can I use these results to justify headcount changes? A: Not yet. At current success rates, agents augment engineers rather than replace them. The time savings are real but require human oversight. Think of it like giving every engineer a very fast, slightly unreliable junior teammate.

Q: What's the biggest blocker to higher success rates? A: Context understanding. Agents struggle with implicit knowledge—the unwritten rules, historical decisions, and cross-module invariants that live in engineers' heads. Better retrieval and longer context windows help, but they're not a full solution.

Q: Should we restructure our codebase to be more agent-friendly? A: The practices that make code agent-friendly (clear naming, good typing, discoverable tests, in-code documentation) are the same practices that make code human-friendly. It's worth doing regardless, but don't restructure solely for agents at this stage.

Q: How do I convince my team to try this? A: Start with a low-risk pilot. Pick a contained module with good test coverage, run agents on a few bug fixes or feature additions, and measure the time from task assignment to merged PR (including review). Compare to historical baselines. Data beats hype.