Seven language models managed a simulated clothing shop in a new benchmark. DeepSeek-V4-Pro, the top-scoring model, cleared the task-specific pass mark in 49% of its trials; a separate scripted shop-operator policy cleared it in 97%.
StoreBench runs an online apparel store called Ostrelle on Medusa v2, a real commerce software platform. The shop itself is simulated: orders, returns, changing supplier prices, customer demand and competing stores move over time. Agents manage it through 29 tools covering tasks such as inventory, pricing, shipping and returns.
The test gives agents limited operations in each 12-hour window. The simulated clock advances with their actions rather than the model’s response speed, so a faster system does not gain extra store time. This lets the researchers test whether an agent can keep up with routine work while adapting to changes in demand and supply.
The benchmark sets a separate pass bar for each task and market seed using scripted policies such as smart-triage. Each trial’s combined score weights business performance at 70%, on-time shipping and handled returns at 20%, and continuity across the task at 10%. Across 11 tasks, three simulated market seeds and three attempts per task and seed, each model ran 99 trials. DeepSeek-V4-Pro averaged 0.700 and passed 49% of them; smart-triage averaged 0.764 and passed 97%.
Smart-triage is a scripted reference within the test, not an independent outside competitor. Its pass rate is part of the benchmark’s calibration context, not a result against retailers or other external systems.
The researchers also ran a separate reinforcement-learning experiment: Qwen3.5-27B was trained on feedback from five tasks and tested on 11 held-out tasks. Under a different, custom harness, its mean score rose from 0.136 to 0.373, but it passed only 3 of 99 trials and stayed below a separate checklist-based policy on all 11 tasks. This was one hand-adjusted run, not a general training recipe, and its scores cannot be compared directly with the seven-model leaderboard.
The results come from one simulated apparel business with modeled demand and scripted competitors. The authors released sample tasks, trajectories and scoring tools, but withheld the full environment and evaluation suite. StoreBench tests sustained store operations in a controlled simulation; it does not establish that current agents can run a real shop.
Sources
Illustrative photo: Vermont Flannel Company retail store, by Tessa Bury, used unmodified under CC BY 4.0. Image source. The benchmark shop is simulated.