A new benchmark finds that language models can often choose a cybersecurity tool without producing a command that is fully correct. Across 24 evaluated open-weight model configurations, KaliBench reports average tool-selection accuracy of 72% in its unrestricted test, but only 22.3% of commands matched the reference exactly.
KaliBench asks a model to translate a short, plain-language security request into one command for a tool in Kali Linux, a distribution used by security practitioners. It scores tool choice separately from the command arguments. A model can name the right program yet still give a misspelled flag, the wrong value or an argument in the wrong position.
The authors built 8,504 request-command pairs covering 1,642 tools, then set aside 5,000 examples for evaluation and 3,504 for training. They say more than half of the initially generated commands contained invented or misspelled options. Their data pipeline filtered candidates with language-model checks, sandbox execution and human review before keeping the final set.
The test also shows what extra information changes. Giving models a list of 20 possible tools raised average tool-selection accuracy from 72% to 95.2%, while exact-command correctness rose only from 22.3% to 28.3%. Supplying the relevant tool documentation raised the exact-match average to 73.1%. These are averages over the paper's open-weight configurations, not success rates in deployed security work.
Because the reference commands were checked in advance, KaliBench can score a model's chosen tool and arguments without running its suggested command. The authors use those deterministic scores as training rewards.
The study covers single-turn command generation from documentation-grounded examples. It does not test whether an AI system can complete a multi-step security investigation or operate safely and effectively in a live environment.
Sources
- KaliBench paper
- Authors' code, data and model releases
- Image: Figure 1 from the authors' paper, licensed CC BY 4.0.