An AI agent can choose an API that fits a request and still fail if that tool is unavailable. Two Texas Tech University researchers describe RATED, a tool retriever that changes its rankings after executions. In a benchmark with simulated outages affecting 30% of relevant tools, it selected unavailable tools in 2.9% of measured choices, compared with 86.2% for ToolLLM’s API retriever. The experiment did not test live agents.
RATED separates matching a request from checking whether a matching tool is working. It first retrieves the 10 tools whose descriptions best fit the user’s query. A second ranking stage considers the query, a metadata-based reliability score and previous execution results for each candidate. That score uses the fit between a tool’s name and description, its catalog source and its documentation quality. After a tool runs, the system receives a success-or-failure signal for that tool. It discounts older outcomes so recent failures can change later rankings sooner.
The researchers built the EcoAgent benchmark from 5,172 tool definitions collected from LangChain, CrewAI, LlamaIndex and n8n. The full benchmark includes more than 210,000 query-tool pairs. Its held-out real-query set contains 6,379 unique requests and 9,946 verified matches; separate synthetic queries were used to train the retrieval model.
On the real-query test, RATED scored 0.618 on mean reciprocal rank, a measure of how near the correct tool appears in a ranked list. That was 52.2% higher than ToolLLM’s 0.406, but below OpenAI Ada-002 at 0.670 and SPLADE-v3 at 0.626. The reported gain is therefore against one baseline, not a lead over every retriever. For this test set, the authors used a language model to verify query-tool matches; human experts reviewed a stratified sample of 200 queries and agreed with the verifier in 94% of cases.
For the separate outage test, the researchers kept a set of benchmark tools unavailable and measured the 3,000 selections after failures began. In the 30% failure-rate setting, RATED’s post-failure success rate was 82.1%; this counts selections that were both the intended tool and still available. Its fault-selection rate—the share of choices that picked an unavailable tool—was 2.9%. ToolLLM recorded 10.5% post-failure success and an 86.2% fault-selection rate. These numbers describe the paper’s injected-failure simulation, not observed service outages.
The authors list validation in live agent runtimes and tests of newly added tools as future work.
Sources
- RATED paper, arXiv record
- Full paper and methods
- EcoAgent code and dataset
- Image: Figure 3 by Chen and Liao, from the paper, CC BY 4.0; unmodified.