Researchers comparing two language models found no consistent recovery advantage for 4-bit or 8-bit versions when tools failed temporarily. In their 20-task test, which version appeared stronger depended on the prompt, which tasks were included and how the test scored each answer.
Quantization stores a model’s weights with fewer bits, which can reduce memory needs. Yuhe Hu compared 8-bit and 4-bit versions of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on 20 synthetic order-management tasks. The test first recorded whether each version could finish a task without errors, then injected a temporary failure at every tool-call position on the task’s required path; an identical retry succeeded. One task could therefore generate several recovery attempts. The authors averaged success across those error positions within each task, then averaged across the tasks included in a comparison. Five prompts varied the instructions for tool use and failure handling.
One Llama prompt showed how the task set changed the result. On each version’s own clean-passing tasks, the task-averaged recovery score favored 4-bit by 17.5 percentage points. Those sets differed: 8-bit passed 19 of the 20 tasks without errors, while 4-bit passed 10. On the nine tasks both versions passed, their recovery scores tied. When the researchers averaged over all 20 tasks and counted a clean-run failure as zero, the full system result favored 8-bit by 28.3 points. The first two comparisons ask how well each model recovers on selected tasks; the last also includes whether it could handle the task before any tool failed.
The scoring rules changed the result again. The main test accepted an arithmetic expression even though the prompt requested a literal amount. Among fault episodes where the Llama model recovered under those tolerant rules, 30 of 38 8-bit episodes submitted an expression, compared with 3 of 20 for 4-bit. A stricter rescore required valid JSON, a machine-readable data format, and a literal numeric amount. Applying those rules to the same logs, without rerunning the models, changed the all-task result from a 28.3-point 8-bit lead to a 15-point 4-bit lead. The strict score also changed which tasks passed the initial no-error screen, so it measures output-format compliance as well as recovery.
Across both tested models and five prompts, the paper found no stable winner: Qwen’s matched-task comparison changed direction across prompts, while Llama’s results also depended on which tasks and scoring policy were used. The study involved 20 synthetic tasks from two related templates, two model families of similar size and specific quantization presets. Its prompts were not controlled paraphrases, and some were written after early results. It does not establish which precision works best for other agents or real-world tools. The paper is a preprint accepted at the NeurIPS 2026 Workshop on Small Language Models for Agentic Systems.
Image: GPU workstation by Wikideas1, CC0 1.0, Wikimedia Commons. Illustrative image; it is not the hardware used in the study.
