In RealtimeBench, Jev averaged 13.51 on a six-game progress scale while games kept advancing during its responses; GPT-6.1-sol averaged 1.51, researchers from Peking University and Samsung Research report. A score of 100 means reaching a game-specific reference target, not necessarily winning. When play froze during model inference, GPT-6.1-sol scored higher in five of six game types. Agents received text descriptions of game states and lists of legal actions, not images. The results appear in a preprint submitted to arXiv on Oct. 14.
Jev scored higher on the live-clock tests
RealtimeBench contains 180 tasks across six game types, including Mario, Snake and Tetris. Each game has 30 tasks: 10 for feedback, 10 for validation and 10 held out for final testing. The agents received a text description of the current game state and a list of legal actions. In the live condition, the previous accepted control stayed active while an agent prepared its next move, so the game could change before that move took effect. In the paused condition, the game froze during inference.
Scores measured the peak progress an agent reached toward a game-specific reference target, normalized to a 0–100 scale. The overall average gave each of the six game types equal weight.
On the paused tests, GPT-6.1-sol, using medium reasoning effort, scored higher than Jev in five game types. With the game running, Jev averaged 13.51 while GPT-6.1-sol averaged 1.51. Their mean response times were 0.52 seconds and 18.97 seconds, respectively. These results compare the models in the authors' controlled game setup; they do not show how either would perform in other interactive settings.
GPT-6.1-sol turned practice failures into guidance
The coaching method has GPT-6.1-sol review Jev's practice failures, test alternative actions in paused or resettable game states, and turn useful corrections into instructions Jev can follow during live-clock play. The researchers call it Two-Timescale Offline Recursive Self-Improvement (TORSI). Jev then acts under the live clock without calling the teacher model. Both models' weights stayed fixed; only the written guidance changed.
The researchers kept test tasks out of the coaching loop. They used separate validation tasks to decide whether to keep each new instruction set, then measured the selected guidance on held-out tests. After two improvement rounds, the selected guidance raised Jev's mean test score from 13.51 to 24.19 across the six game types; each score was averaged over three runs. The paper also reports higher average scores after applying the same guidance to three other models without retuning, though one model regressed in two game types.
The results remain limited to controlled games
The authors say the benchmark gives agents structured text states rather than raw images, so it does not test visual perception. Their experiments cover six game types and two coaching rounds, and the transfer tests reuse the same environments. TORSI also depends on saved or resettable states for offline practice. The paper does not establish performance in physical robots, unfamiliar environments or longer-term deployment.
Image: Photo by Evan Amos via Wikimedia Commons, public domain. Illustrative controller, not a RealtimeBench setup. Image details.
