Researchers testing small, locally served AI agents found a measurement trap: a server can refuse a request listing available tools before the model runs, while the evaluation software keeps the error as a generic assistant message. A later analysis could count that message as a failed model call and report a zero score, even though the model never produced an answer.
A coding agent uses tools to inspect files or run commands. To test whether it can request one correctly, an evaluation program sends the model a list of available tools through a serving system such as Ollama, then checks the response. The serving system must first accept the request and translate the tool descriptions into a form the model can read. A failure at that stage says nothing about the model's choice of tool.
Ollama rejected tool requests before the models ran
Lijuan Tang and Yuemeng Zheng used a ReAct-style extension of LOCA-bench to test the first step of local coding-agent tool use. With Ollama 0.30.8, requests for Phi-3 and Gemma-3 were rejected with HTTP 400 across all eight task instances in the main test. The harness kept a generic error message in the assistant's trajectory but lost the structured rejection type. A downstream analysis that treated each message as a model turn would count those cases as a zero-percent valid-call rate; the rate is actually undefined because the models did not run.
The serving system, rather than the model weights, caused this result. When the researchers sent the same tool request to llama.cpp with the same GGUF model weights, Phi-3 and Gemma-3 returned text responses. In single-request probes, vLLM rejected the tool request by default until tool-choice and parser options were enabled; SGLang accepted it but returned the call as text without a tool-call parser. Those checks covered only Qwen-0.5B and Phi-3 and did not reproduce the full agent tests on those systems. The particular Ollama rejections are tied to the tested release and may change with its model templates.
Prompt format and counting method shifted valid-call rates
Even when Ollama accepted a request, the measured rate changed with how tools were described. For Qwen2.5-Coder-3B, adding a plain-text tool list and call format to the prompt while retaining the same tool-request channel raised the average valid-call rate across eight task instances from 23% to 82%. That rate measures correctly formatted, available-tool calls among turns the model produced; it is not a measure of completed coding tasks. A uniform text-only protocol, which put the tool list and call format in the prompt instead of a tool-request field, was not always better: for Llama-3.2-3B, it lowered the per-instance rate from 82% with native tool requests plus a hint to 44%.
How the turns were counted also mattered. For Qwen2.5-Coder-0.5B under the text-only protocol, pooling all turns yielded an 85% valid-call rate, while averaging the eight task instances equally yielded 34%. One 41-turn loop dominated the pooled figure as seven other instances failed quickly. The authors repeated the server-rejection pattern on a second task and a small single-turn HumanEval probe, but the number of instances was too small for stable model rankings.
The authors recommend recording server refusals and empty responses separately from model non-calls, and reporting the serving version, tool-call parser and per-instance results alongside the model name.
Sources
- Tang and Zheng's research paper (posted September 22, 2026).
- Released code, configurations and results.
- Image: Server racks by CSIRO, CC BY 3.0. Illustrative; these are not the researchers' test machines.
