Researchers from Fudan University, Tencent’s Hunyuan team and Tsinghua University have introduced ExplorationBench, a test of whether AI systems can choose useful experiments, learn unfamiliar rules and apply them to new problems. The two test worlds use software to run programs and check proofs, so answers can be scored directly without an AI judge.
Static benchmarks show whether a model can solve a task; they do not reveal how it gathers evidence when the rules are unknown. ExplorationBench builds two rule-based “alien” worlds that break familiar assumptions: AlienCode changes how programs behave, while AlienLogic changes the rules for proving statements. Each has 70 held-out tasks and, respectively, 31 and 24 hidden-rule targets.
Ten AI systems started with the same flawed manual and worked examples, then chose probes over four rounds. After each round, a separate tool-disabled copy reported the rules it believed and answered the held-out tasks, with each task answered three times. The researchers could therefore compare what a system said it had discovered with whether it could use those rules. The models’ weights were not updated, and the paper says no persistent memory was allowed.
The arXiv v1 paper reports that the best of three independent AlienCode exploration paths reached 87.6% accuracy after four rounds, from a maximum of 15.7% before exploration. The authors’ current project site now lists 89.0%. The paper reports that when a system correctly stated every rule needed for a task, it solved 70.9% of 647 task-and-exploration-run cases; the site now lists 73.4%. The site’s current results asset refers to a later findings_v2.py build, but the authors have not said whether the new figures reflect reruns, a correction or recalculation; arXiv still lists only v1. The two versions should not be treated as interchangeable.
The paper also reports large differences between repeated exploration paths: for one system and budget, final scores differed by as much as 72.8 percentage points, and six of 30 AlienCode paths ended at least three points below an earlier milestone. The paper’s headline score is the best result among three independent runs, not a typical-run estimate. Systems used their own API’s highest reasoning setting, so the test was not compute-equivalent across providers.
ExplorationBench makes evidence gathering and rule use measurable in a controlled setting, but it does not show that today’s systems can make discoveries in laboratories or open-ended research. The task data are not public; the project site says outside evaluations are run by request. Reproduction and generalization beyond these synthetic worlds remain open questions.
Sources
- ExplorationBench paper, arXiv v1
- Official ExplorationBench project site
- Current project results data
- Current project configuration and evaluation access
Illustration: Figure 2 from the paper, used under its CC BY 4.0 license.