A new benchmark asked 12 language models to explore simulated worlds with altered physics and decide for themselves what to investigate. Only one model recorded any discovery that was both correct and supported by experiments, according to a paper posted Oct. 27: GPT-6-Astra, with a reported discovery rate of 16.7%.
Illustrative image: a Newton’s cradle in motion. Photo by Arigoodwoman, via Wikimedia Commons, CC BY 4.0.
The result comes from LongChang, a research framework built around simulated scenes in which familiar objects behave under hidden, changed rules. It measures a narrow skill: noticing something unexpected, choosing a question and testing an explanation without being assigned a research task. The other 11 models scored zero on the discovery measure.
The models had to choose what to question
LongChang’s base set contains 15 scenes in MuJoCo, a physics simulation engine: one control scene with ordinary physics and 14 scenes with one altered law each. The models began from the same initial states and could interact with balls and boxes, observe what happened, and build experimental tools. Each exploration had a limit of 100 turns.
In one example, a red ball began moving again without a new push. GPT-6-Astra put aside its planned collision test, measured the delay in the simulation, then tested whether pushing a blue ball also reset the red ball’s motion. Gemini-3.7-Flash observed related movement but continued its planned tests, leaving the anomaly unexplained.
A correct guess was not enough
LongChang counted a discovery only when a report identified the altered law and its relevant objects and conditions, and the recorded experiments supported the claim. The authors’ project page says each model ran each scene three times. That gives GPT-6-Astra 42 altered-physics trials: its 16.7% discovery rate represents seven trials passing both checks. Its separate scores were 19.0% for law correctness (eight trials) and 97.6% for evidence support (41 trials). Claude-Opus-5, for example, scored 16.7% on correctness but zero on evidence support.
The ordinary-physics control also tested overclaiming. Across models, 36.1% of reports were rejected for falsely claiming an altered law or offering an unsupported explanation of normal behavior. That matters because an agent that proposes many hypotheses can appear to reason scientifically while overlooking whether its tests rule out alternatives.
LongChang tests models against preset physics rules
The benchmark gives researchers a known answer key: the altered laws are defined in advance, so a report can be checked against them. Real scientific questions often lack an established answer, and the authors say extending the framework to those settings needs more research. An LLM designer specifies each scene’s objects, geometry and altered law; separate LLM agents review scenes and tool designs, then judge reports against the preset laws and saved experiments.
The paper is a preprint.