A team including Google Cloud AI Research and the University of Wisconsin–Madison says its new agent workflow improved software-optimization results by updating which research ideas coding agents pursue as experiments produce evidence. On five CPU tasks, the Agentic Idea Manager (AIM) scored 67.0 on AutoLab’s normalized 0–100 scale, 1.6 points above the strongest idea-search baseline, ScientistOne, according to a September 29 preprint.
That score is not an accuracy rate. On the five system-optimization tasks, invalid code or code no faster than the benchmark’s deliberately weak starter implementation scores zero. The human-written reference implementation anchors the scale at 50; faster-than-reference code can score above 50, up to 100. AIM keeps a pool of proposed approaches, groups related ideas by research direction and ranks the directions and individual ideas using experiment results, evidence gaps, novelty and implementation lessons.
The framework then chooses whether to explore an uncertain direction or exploit a better-supported idea, and sends selected ideas to separate solver agents that implement and test them. After each run, a Solution Auditor checks that the code meets the task, actually tests the selected idea and has not gamed the benchmark. The system uses that audit to update its idea pool and adjust how many solver branches run in parallel under the remaining experiment budget.
Across five additional model-development and CUDA tasks, AIM averaged 55.8, 4.9 points above ScientistOne’s 50.9. That mean combines task scores separately normalized to 0–100: CUDA uses runtime anchors, while model-development scores normalize each task’s quality metric between its baseline and reference results. AIM did not lead every test: on Data Selection IFEval, it scored 61.8, while the solution-search system AdaEvolve scored 82.7. The authors report AIM had the highest average on three of the five tasks in this group.
The study used ten fixed AutoLab benchmarks in containerized environments. All compared agents used the same Gemini-3.1-Pro-Preview backbone, and each method-task result is averaged across three runs. These are software-optimization and model-development tasks, not open-ended scientific research. The authors say no generated solution was deployed in a real-world system and that human review remains necessary before consequential use. The preprint suggests that explicit idea management can help on bounded coding tasks; whether it helps other models or actual research workflows remains untested.
Sources
AIM: Agentic Idea Management for Automated Research (full paper)
Authors’ project page and evidence explorer
Lead image: Figure 2 by Hyeong Kyu Choi et al., used unchanged under the Creative Commons Attribution 4.0 license.