A new benchmark asks whether language models can work out what Python software actually does when tests run across a full repository. In a September 23 preprint, researchers at York University and the University of Texas at Dallas report that GPT-5.4, the strongest of five tested models, answered 37.71% of SWE-Flux questions exactly. The result measures one demanding task, not coding ability overall.
Most repository question-answering tests ask models to find or explain information in source files. SWE-Flux instead asks about behavior during execution: which branches and functions ran, what values variables held, and what changed across a set of tests. Its dataflow questions, for example, ask where a value came from and which later operations used it.
The benchmark contains 480 questions drawn from 12 Python repositories, including Keras, Matplotlib, SymPy and yt-dlp. Researchers ran each test with instrumentation that recorded events such as executed lines, function calls, return values and exceptions. Scripts then derived the expected answers from those traces, rather than relying on manual labels or another language model to judge them.
During evaluation, the models could inspect the repository, test files and question, but they could not run the code themselves. They had to predict a structured answer, which the researchers compared with the trace-derived answer using exact matching. GPT-5.3-Codex scored 33.75%, behind GPT-5.4. The largest weakness was dataflow: even the top model scored 6.78% on single-test questions and zero on the multi-test version.
The researchers also inspected 299 incorrect GPT-5.4 answers; 93% contained substantive mistakes in execution reasoning, rather than just formatting problems. Errors included following paths that a test never took, losing track of state, and inferring behavior across a test suite from only part of its evidence. This helps explain why a model can give a plausible code explanation yet miss what a particular run actually does.
The findings are limited to five models, exact-match scoring and 12 Python projects. They do not show how models perform when allowed to execute code. A useful next comparison would test coding agents with execution tools across more repositories and languages, separating what they infer from source from what they verify by running tests.
Sources
- Taherkhani et al., “Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark,” arXiv:2609.28449 (submitted September 23, 2026; preprint)
- SWE-Flux benchmark code and question data (MIT license)
- Lead illustration: Figure 1 from the authors’ paper, reused under its CC BY 4.0 license.