Meta says Muse Wonder Full captured 92 ExploitGym flags, but only 21 used each task’s intended vulnerability. Its model card says 21, not 92, is comparable with the benchmark’s published ranking.
ExploitGym checks how a flag was reached
In ExploitGym, each test puts a secret flag beyond an agent’s authorized access. Retrieving the flag shows that the agent got the test system to run code without permission, but does not by itself qualify as a benchmark success. A second-stage evaluator checks whether the exploit used the vulnerability specified by the task rather than a different weakness or shortcut. That check determines whether the capture counts as success in the published ranking. The paper says these controlled tasks use vulnerabilities found in real software.
Meta says 21 captures used ExploitGym’s intended vulnerability
Meta’s October 12 model card reports 92 flag captures. A GPT-cyber verifier found that 21 used the intended vulnerability, and Gemini 3.1 Pro independently confirmed that result; the other 71 came through unintended vulnerabilities. Because ExploitGym’s published ranking counts only captures using the task’s intended vulnerability, Meta identifies 21 as comparable.
The card provides aggregate counts, not case-by-case traces showing how the 71 other captures occurred.
Sources
- Meta Muse Wonder Model Card, October 12, 2026.
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, arXiv, May 11, 2026.
- Meta’s Muse Wonder and Muse Wonder Full announcement, October 12, 2026.
- Lead image is illustrative; it does not depict Muse Wonder or the ExploitGym tests. Cybersecurity.png by jaydeep_ via Wikimedia Commons, CC0 1.0.