An October 9 preprint from the Alan Turing Institute reports that selected language models reached English Wikipedia targets at rates comparable to a small human benchmark. Among successful runs, human players generally reached the target in fewer steps. The test was link-only navigation, not open-web search.
Researchers compared language models, sentence-embedding systems and a statistical text baseline on ten graphs: five Wikipedia language editions and five subject areas built from OpenAlex academic-paper references. Each agent started on an article and had to reach a named target using only links on its current page; it could not look ahead or work backward from the destination. Every agent could see the target summary. In their best settings, embedding agents chose the outgoing link whose article summary most resembled the target summary, while language models selected among link titles.
On the English Wikipedia graph, the reasoning version of GPT-o3 mini had the highest success rate among selected models. The paper also describes GPT-4o, QwQ:32B and DeepSeek-R1:70B as comparable to people. Success meant reaching the target within 100 steps. In successful games, human players generally reached it in fewer steps than the model agents.
Nine people played 109 games under a ten-minute limit. They self-reported success, and the researchers say the games were not run under strict experimental conditions. The model agents had a 100-step cap, and the model families received different inputs. The paper says resource limits prevented some agents from playing all 10,000 generated pairs; GPT-4o was limited to 100 games because of cost.
Results changed with the graph. The researchers tested English, French, Chinese, Russian and Bengali Wikipedia, along with OpenAlex graphs in biology, medicine, physics, social sciences and arts. Model rankings did not stay constant across languages and subjects, and the human comparison covered English Wikipedia only. Some embedding methods also outperformed many of the language models on parts of the benchmark.
Sources
- Hickey, Knight and Bishop, “Can Language Models Play the Wikipedia Game?” arXiv:2610.13948v1. The lead image is Figure 2 from the paper, reused unmodified under CC BY 4.0.
- The authors’ public code repository for graph preparation, agents and report figures.