An Oct. 6 arXiv paper from ETH Zurich introduces Agentic AutoRAG, a tool that tunes retrieval-augmented generation (RAG), which searches documents for passages before a language model answers. The researchers repeated each method’s 30-trial search with 10 independent seeds and report median accuracy (the middle result across runs) at each API cost. On an exam generated from healthcare research documents, Agentic AutoRAG answered 77% of questions correctly at $0.000741 in API costs per question; a warm-started Bayesian optimizer peaked at 71.5% at $0.001284.
RAG pipelines can fail because the document search misses useful evidence or because the answer model mishandles passages it did retrieve. After each trial, Agentic AutoRAG checks which kind of error occurred. One agent summarizes those failures; a second uses that summary, plus public model rankings and prices, to choose the next setup. It tests different ways to split documents, find and reorder passages, and generate answers.
The cost test used 230 PDFs and 20 page images, mostly biomedical research articles and medical studies. The researchers generated and froze an exam from those files instead of using UniDoc-Bench’s labeled questions. Before its 30 counted trials, the baseline received 30 earlier trials on the same document collection without counting them toward its search budget; the agent had 30 trials in total. At the same 71.5% score, the agent’s median result reached that level for $0.000288 per question, about 22% of the baseline’s $0.001284 cost.
A separate evaluation used 300 questions from each of three benchmarks, all held back from the tuning runs. The questions required systems to combine evidence from multiple passages. An LLM judge compared each answer with a reference response; the paper reports higher scores than all baselines. The healthcare exam was generated from the document collection and scored by an LLM, with no check against human reviewers. Reported API cost includes generation and query expansion, but excludes embeddings and reranking run locally. These tests do not establish clinical accuracy or patient outcomes.
Sources
Image: Illustrative stock photo, not part of the study. Photo by Vitaly Gariev under the Unsplash License.