In a test of AI-written fixes for C and C++ vulnerabilities, changing only the compiler's language-standard setting made the same patches compile 1.8 to 2.7 times as often. No patch changed. The result calls into question a common shortcut for judging whether an automated security fix works: counting how many candidate patches build successfully.
Researchers at the University of Southern Mississippi tested three open-source code models on 203 vulnerable functions drawn from the Big-Vul dataset. Their evaluation checks individual functions outside the programs they came from. An isolated function may lack definitions supplied elsewhere in its program; the dataset's extraction can also strip its return type. Either can prevent compilation without the generated edit being the cause. The team classified about 64% of compile failures across the three models as attributable to the test setup rather than the model output.
The compiler-setting experiment held all generated patches fixed. Switching from the harness's default setting to the older gnu89 C standard raised the overall compile rate from 5.45% to 10.25% for CodeGen-350M, from 3.45% to 8.83% for DeepSeek-Coder-1.3B, and from 3.32% to 8.90% for DeepSeek-Coder-6.7B. The flag changed how C candidates were checked; the C++ candidates were compiled the same way under both settings. The older C standard permits some constructs that later standards reject; it did not make a security repair more effective. In a separate feedback test, an instruction-tuned DeepSeek-Coder-1.3B's compile rate rose from 3.9% to 9.4%, while inspection found some newly compiling outputs that deleted code or replaced a function with a placeholder rather than fixing it.
Comparing an entire generated function with the human-written fix was no reliable fallback in these tests. An unchanged copy of the vulnerable input scored higher on the whole-function CodeBLEU similarity metric than any tested model, because most lines in a correct fix are usually unchanged. The researchers examined an edit-only score, diff_F1, that gives a no-op zero credit and catches some deletion-style outputs. It also credits other bad patches, so they present it as a screening tool, not a measure of whether a vulnerability is removed.
The tests cover small and midsize models, one dataset and isolated C/C++ functions; they do not establish how larger models or full-repository repair systems perform. The team did not run exploit or execution tests on the patches. In this setting, a reported compile rate needs its toolchain and test harness alongside it, and a claim of a repaired vulnerability still needs a check of the program's behavior.
Sources
- Original vulnerability-repair metrics study, posted September 22, 2026; see Methods, Results and Limitations. The authors provide code and data.
- Lead image: illustrative C++ code screenshot by Micke, CC BY-SA 3.0, displayed without modification. It is not a patch from the study.
