A keyword-based test gave two small Spanish cybersecurity AI models nearly identical tool-use scores. But in a stricter test, the larger model never produced a software-usable tool call.
Independent researcher Juan S. Santillana compared two VectraYX models with 662 million and 1.1 billion parameters. The study’s B4 benchmark scored their tool use at 0.660 and 0.650. But its keyword matcher could award credit when an answer merely mentioned a tool name in ordinary prose; it did not check whether the model emitted a structured call that software could use.
A stricter check replayed examples from the models’ own tool-training data and required a valid JSON call with plausible arguments that differed from the originals. The 662-million-parameter model passed all six examples. The larger model passed none of four to six examples at each checkpoint tested, even after a dedicated fine-tuning phase using about six billion tool-use tokens.
After diagnosing the failure, the author fine-tuned the larger model for 2,202 steps, a run reported to take about 3.3 GPU-hours. On an author-built set of 238 prompts about entities absent from training data, 166 required a tool call. A full pass on those 166 required the right tool and complete arguments containing the requested unseen detail or expected shell-command family. The repaired model met that standard on 53.6% of prompts, compared with 42.8% for the smaller model (paired p=0.0039).
The evidence is narrow: this is one model pair, not a controlled experiment isolating the effect of training data, and the smaller model’s training stopped at 64% of its planned schedule. The repaired model represents one tuning recipe and one checkpoint. The author-built test did not cover unseen tools or multi-call sequences, and both models sometimes called tools when none were needed. Producing a well-formed call also does not establish that a cybersecurity action is safe or correct.
Image: Alicia Christin Gerald / Unsplash. Illustrative developer photo; not part of the study.