In a synthetic coding benchmark, six tool-using coding agents made more mistakes as two code components grew from five to 200 lines. They had to move code shared by both into a common parent component while preserving the behavior unique to each, researchers report in an Oct. 7 preprint.
The paper introduces CABRA, a benchmark that generates code-editing tasks from graphs of functions that call one another. Across 6,840 tasks, the researchers varied one challenge at a time: following a change through connected functions, searching among extra functions, tracing which code runs, and carrying out more instructions. They compared eight language models without tools and six coding agents with tools in GitHub Copilot’s public harness.
Tools helped agents handle several larger tasks. As task size grew, standalone models’ accuracy fell, while agents often stayed near-perfect on function traversal, search and runtime-resolution tasks by using tools such as grep, scripts and code execution. Instruction-following remained a harder axis. Agents’ logs also showed more reading and analysis on larger tasks, even when the amount of editing did not rise in step.
The authors then examined public mini-SWE-agent traces from SWE-bench Verified, a benchmark of repository bug-fixing tasks. Read and Analyze tool-call counts had a stronger relationship with accuracy than the number of lines changed, although both correlations were weak: −0.20 for understanding calls and −0.159 for edited lines. Because real tasks vary in many ways at once, this trace analysis cannot show that reading more code causes an agent to fail.
CABRA’s harder task asked an agent to compare two code components, called classes, and move their shared code into a parent class while leaving behavior-specific lines in each component. The researchers made equivalent code harder to recognize by renaming variables, reordering equivalent operations and splitting calculations across lines. As each component grew from five to 200 lines, agents’ accuracy fell. The paper says agents preserved behavior in more than 90% of cases, but often failed the refactoring requirement by leaving shared logic behind in the child components.
CABRA is synthetic: it shows agents can miss shared-code refactoring as components grow, but not how often that failure happens in ordinary repositories. Its code and tasks are available in the authors’ repository. The arXiv preprint supports testing code-understanding tasks separately from edit volume.
Sources
- CABRA paper, arXiv:2610.10610v1, submitted Oct. 7, 2026.
- Microsoft Research publication page.
- CABRA code and task repository.
- Figure 5, reproduced unedited; paper authors, CC BY 4.0.