A study of AI coding agents found that text descriptions helped when the target source file was hidden, raising the average fraction of tests passed from 8% to 71% across 11 selected cases. When agents could inspect the source, however, adding descriptions did not significantly improve issue fixes.
The researchers built a round-trip benchmark: a model describes a Python file in text, then tries to rebuild it, and the original tests measure whether the description preserved enough detail. An automated search improves the prompt that creates those descriptions. The benchmark and optimizer are research prototypes, not a coding-agent product.
For the source-hidden test, the team selected 11 single-file Python issues from SWE-bench Verified and ran each three times. The score is the average fraction of tests passed per issue, not the share of issues solved. Here, the optimized descriptions stood in for source files the agent could not read.
The result changed when agents could inspect the source. On SWE-ContextBench Lite, Gemini 3.8 Flash resolved 33 of 58 paired tasks with issue text and repository code but no added documentation, 29 with an approximately 80-word file summary, and 30 with retrieved past-task context. The official Docker evaluation found neither addition significantly changed results. A task counted as resolved only when the benchmark’s required tests passed; two of the benchmark’s 12 repositories supplied no scorable task.
Smaller tests with Qwen 3.6 and Gemini 3.1 Pro showed the same general pattern when source was present. The authors traced some failures to agents rewriting a whole file instead of making a narrow fix, but found that summary length alone did not explain the penalty.
The evidence has limits. The source-hidden test used 11 selected single-file Python cases; the largest source-present run used Gemini 3.8 Flash, while stronger-model replications were confined to Django tasks. The work is an arXiv preprint. Its next important test is at repository scale, where all the source may not fit in a model’s context window—the condition where descriptions are meant to help.