A new benchmark found that AI agents can produce digital LEGO designs that pass simulated build checks, but people could still identify the human-designed models. Five raters familiar with LEGO design were asked which model in each pair was human-made; they chose it in 323 of 360 ratings. The pairs matched by piece count, not by prompt or subject, so the study does not compare human and AI answers to the same request.
The Oct. 8 preprint, BrickBench: Evaluating Agentic Brick Design, tests coding agents asked to turn written prompts into digital brick assemblies. Its 300 prompts are split across three settings: designs using up to 400 pieces; set-scale builds using 400 to 4,000; and alternate builds restricted to the 783-piece inventory of LEGO set 10698. Eleven general-purpose coding agents take the full test; two task-specific models are compared in the smaller setting.
The agents work inside BrickAgent, an environment that lets them search a library of brick parts, place pieces using compatible connectors, combine subassemblies, render a design and run checks. They can inspect the reported errors and revise their code. The task therefore measures what an agent can do with tools that expose part geometry and give feedback, not what a language model can infer from a prompt alone.
Those tools make a large difference to whether a design passes. GPT-6 Astra’s validity score was 100% with BrickAgent and 40% when researchers removed its build and validation tools. GPT-5.6 Luna produced one valid assembly out of 300 without those tools, less than 1%. With the environment, Astra also satisfied 95.4% of the benchmark’s visual question checks about whether its builds met the prompts.
“Valid” here has a specific, limited meaning. A design must meet its piece-count rule, avoid collisions and remain at rest in a physics simulation. The simulator treats connected groups as rigid bodies and does not model the holding force of LEGO studs or the movement in some joints. A design can pass while a physical version would sag or come apart; the authors describe their test as necessary, but not sufficient, for real structural stability.
For each agent design in the up-to-400-piece and set-scale tasks, researchers chose a human design from the BrickNet dataset with a similar piece count, then asked raters to identify which one had been made by a person. BrickNet had no human designs for the benchmark prompts, so paired examples could depict different subjects. The authors cite part use, proportions and surface detail as areas where agent designs lagged.
The authors have released BrickBench and the BrickAgent environment for others to inspect and test.
Sources
- BrickBench: Evaluating Agentic Brick Design (arXiv v1, Oct. 8, 2026)
- BrickAgent source repository
- Illustrative photo: “Playing with Lego bricks” by Nenad Stojkovic, reproduced unchanged under CC BY 2.0. The photo is not from the study.
