AI agents can reuse saved procedures, checklists or code as “skills.” A six-benchmark study found that 21 such skills improved performance on practice tasks compared with the same agent without a saved skill. Only five kept all of that gain on held-out tasks of the same type; 13 kept part and three kept none.
The study compared five methods that revised a shared skill using task feedback with a one-shot skill written once and left unchanged. Within each benchmark, the methods used the same model, agent setup and task split, and test tasks stayed hidden until each method chose its final skill.
The held-out tasks were new examples of the same kind of work, such as different spreadsheets or code repositories. The result measures generalization within a task type, not whether a spreadsheet skill transfers to coding or another domain.
The authors found a recurring failure pattern: a skill could hard-code one spreadsheet’s column names or output file, or turn a fix for one failure into a rule for every task. Those edits could help on practice examples while hurting other tasks of the same kind.
The paper also tests Generalizable Skill Optimization (GSO), which improves a guide for writing skills instead of preserving one shared procedure. For each new task, the agent uses that guide to write a fresh, task-specific skill, checks its work and may retry once; it then discards the skill. GSO scored highest on all six benchmarks, 4.3 to 22.5 points above the best existing method on each. But it also makes an extra model call for every test task, so this compares complete methods rather than isolating the effect of generating a fresh skill.
The findings are from an arXiv preprint using one model and six benchmarks. Test sets ranged from 17 to 40 tasks, so one task could move some benchmark scores by 2.5 to 5.9 percentage points. The authors say results on other models and transfer between task types remain open questions. For teams building self-improving agents, the practical test is whether a saved skill still helps on held-out examples of the task it was built for.
Sources
- Original paper: Do Self-Evolving Skills Generalize to Held-Out Tasks?
- Illustrative photo, not part of the research: Laptop coding programs by Tirza van Dijk via Unsplash/Wikimedia Commons, CC0 1.0; license.
.jpg){kind=link}