Ankit Sonthalia and colleagues introduced BOTTLED, a benchmark that asks AI agents to build reusable solutions for millions of similar items instead of calling a large language model (LLM) for every item. In 48 of 60 runs, the agent-built solution scored below the lower end of the 95% confidence interval estimated for that model’s direct answers.
Each agent received all task items but not their correct answers, and had up to 10 hours and a five-million-token budget. The budget counts generated output twice as much as uncached input, and cached input one-tenth as much as uncached input. Agents could write a program, train a smaller model, or combine methods. All 10 language models served as the agents and were tested on three tasks: extracting product attributes from listings, matching shopping queries to products, and detecting AI-written text. Workloads ranged from 2.62 million to 5.62 million items, with two runs per model and task.
Each model answered a uniform 1,000-item sample one by one. The researchers resampled those items 2,000 times to estimate a 95% confidence interval, a range that reflects uncertainty in a score estimated from the sample. Agent-built solutions were scored on the full workload, with each reported score averaged across two runs. For each task, the authors used GLM 5.3 Flash to label sample items within the same five-million-token budget as an agent run, then fine-tuned two small models on those labels. In a separate comparison, the stronger small-model baseline scored higher than 31 of 60 agent runs, and both small-model baselines scored higher than 21.
The shopping query-product task shows the cost and score trade-off. For Claude Opus 5, the reusable solution scored 0.499 macro-F1, while one-by-one answers scored 0.609. Macro-F1 averages a separate F1 score for each of the four relevance labels, giving them equal weight. For each label, precision is the share of items predicted to have it that actually do; recall is the share of items with that label the model finds. F1 combines the two and falls when either is low. At $26.28, the reusable solution cost about 99.8% less in API-token fees than $17,273 estimated by the authors for direct Claude Opus 5 calls across all 2.62 million items. It retained about 82% of the direct-answer score.
The paper was posted to arXiv on Oct. 6 as a v1 preprint. Its reported 60-run results include incomplete submissions: missing answers were replaced with task-specific fallback responses. Of the 21 runs that reached the token limit, eight returned incomplete output. With three tasks and two agent runs per model-task pair, the benchmark does not establish reliability or total operating costs in a live service.
Sources
- Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? (arXiv:2610.08775v1)
- scikit-learn: F1 score averaging by label
- NIST Engineering Statistics Handbook: bootstrap resampling and uncertainty intervals
Figure 2: Ankit Sonthalia et al., arXiv:2610.08775v1, licensed under CC BY 4.0.