AI agent software makes small choices, from selecting which language model should handle a step to deciding whether retrieved text is useful. A paired study found that two decision models were no better than chance at routing requests between language models.
System-1 models, as the authors call them, return probabilities for a fixed set of choices in a single pass instead of generating a full text reply. In an October 1 arXiv preprint, Jiawei Li compared open-weight Laya with hosted Jev on 11 types of agent decision, using paired, byte-identical inputs drawn from 18 public sources. Jev scored higher on nine types; the models tied on routing and on checks of whether retrieved passages were relevant.
One part of the study asked whether a decision model could screen out requests it judged clearly benign before a larger language-model safety review. At the tested threshold, Jev sent 65.6% of requests to the reviewer. The paper’s first cost estimate claimed 23.9% savings; after counting the screen’s own tokens, the author revised the projected saving to 4.3%. In held-out threshold checks, the screen introduced at least one additional miss in 47% of the data splits. These are modelled results, not savings observed in a deployed service.
The estimate shows why a screening model has to be evaluated inside the full workflow: its own cost can eat into the savings it is meant to create. But this single-author preprint tests one version of each system, zero-shot, on public, mainly English data. Its author says the routing result applies to the labels in this test and does not establish a general limit on model routing. Held-out agent traffic with independently checked safety labels would be needed to assess production performance and total costs.
Sources
- Jiawei Li, “Fast Models, Slow Evidence,” arXiv:2610.02267v1 (October 1, 2026). Methods, paired results, safety-screen cost audit and limitations.
- Author’s evaluation repository with cases, raw outputs and analysis code.
- Illustrative lead image by Markus Spiske, via Wikimedia Commons, CC0 1.0. The image is illustrative and is not from the study.
.jpg){kind=link}