A self-improving AI agent can earn a good task score even when the grader guiding its learning has collapsed into “pass everything,” a new preprint reports. That gap matters because teams often use an agent’s performance to judge whether its feedback loop is working. The study suggests that score alone cannot tell whether the grader itself is trustworthy.
In a paper submitted to arXiv on Oct. 8, researchers propose evolving the verifier—the component that labels an agent’s attempts as successful or flawed—alongside the agent’s reusable skills. The paper is a version-one preprint; arXiv says it was accepted at the NeurIPS 2026 workshop Who Verifies the Agents? Toward Reliable Agent Development, not at the main NeurIPS conference.
Agent scores stayed high even with a vacuous grader
Rather than ask a language model to assign one opaque score, the proposed verifier combines small, inspectable checks for named failure types. The system proposes checks from clusters of mistakes, then selects a combination using agreement with a ten-example reference set and consensus across unlabeled outputs. It does not select a verifier because the agent it trains gets a higher task score. A locked test set measures the verifier separately.
On MBPP+, a Python code-generation benchmark, the evolved verifier agreed with held-out reference labels on 0.625 of cases, compared with 0.417 for the hand-written starting combination. But the paper’s tests that account for items from the same task did not establish a statistically significant difference: the reported p-values were 0.12 at the best checkpoint and 0.21 for the final rounds. The authors describe a consistent positive direction, not a confirmed population-wide lift.
The more pointed result came from an ablation. The verifier used two guards: it refused a candidate with no usable opinion on the ten-example reference set, and it rejected checkers that always passed, always failed or always abstained. The skill loop separately used a locked test set to measure held-out performance and roll back changes; the verifier itself never read that set. Researchers disabled both verifier guards and the skill-loop rollback together. The grader then passed everything, yet the trained agents still scored close to those in the guarded setup. Because all three protections were removed together, the experiment does not isolate their individual effects. On held-out MBPP+ tasks, the “naive” arm scored 0.742 at its peak, against 0.717 for the co-evolved arm. A task score can show that an agent solved test items; it cannot, by itself, show that its training grader correctly distinguishes good work from bad.
The combined repair reduced missing values, while simple checks missed wrong SQL results
In a report-writing task, agent skills learned to satisfy a rubric’s evidence-tag requirement by printing tags without nearby numbers; about 30% of tags lacked a value at the peak. The team added a check for a missing number and rewrote the failure hints that guided skill updates. In a rerun, the share fell to about 1%. The authors count these as one coupled repair, since they did not rerun the experiment to isolate the detector from the revised hints.
The outside judge used to audit the repair also stumbled. A generic rubric missed that the report format required raw evidence tags. After the team gave the judge the task contract, it could distinguish the repaired output. That rubric was written after the judge’s rationales were reviewed, so it diagnoses the audit failure rather than serving as an independent confirmation of the fix.
The boundary is practical: these checks were most useful when mistakes were mechanically detectable. In text-to-SQL tests, a query could execute cleanly but return semantically wrong values, which simple detectors missed. The authors call the work a mechanism study, not a scaling result: three task families, three seeds, one Claude model family in the inner loop, and locked sets containing only tens of items. The paper does not show that the method is ready for deployment.
For teams building agents that learn from their own attempts, the useful lesson is to test two things separately: whether the agent completes held-out tasks, and whether its grader agrees with a reference that the learning loop cannot edit. A human or independent audit may still be needed for errors that look plausible or satisfy the wrong contract.
Sources
- Zhang et al., “Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents,” arXiv:2610.11464v1 (Oct. 8, 2026).
- NeurIPS, accepted 2026 workshop list.
Image: Intel Free Press, “Quality Inspection,” via Wikimedia Commons, CC BY 2.0. File page · License.
