Mem0 researchers have released DolphinBench, an open test of whether AI agents can use details from past conversations while completing tasks in simulated apps. Instead of asking an agent to recall a fact, the test asks it to act on one without a reminder. The team's results show why task accuracy alone can be misleading: stronger memory can raise the total cost of a run.
In one test, a fictional startup chief mentions a preferred coffee shop in an earlier conversation. Months later, the chief asks for a pickup order but does not name the shop. An agent passes only if it uses the earlier preference and places the correct order. A direct question about the coffee shop would have given away that it needed to search its memory.
Each task depends on earlier context
The researchers generated long histories for three fictional workers, each with roughly 500,000 tokens of user messages. Their 600 tasks span simulated tools such as email, calendars and project apps. For each task, they ran an agent twice with the relevant history and twice without it. A task was accepted only if both informed runs passed and both uninformed runs failed. Grading checks the actions and their results, rather than a written answer alone.
The dataset and evaluation code are public. A memory system is judged on the percentage of tasks fully completed, the total cost of processing history and running tests, and the median time per task. Those measures make a faster or cheaper system visible even when its task score is lower.
A higher score carries a cost
With the same Hermes agent framework and GPT-5.6-Luna model, Mem0 completed 70.67% of tasks against 65.67% for the framework's built-in memory. Median task time fell from 44.35 to 37.69 seconds, but total cost across history processing and testing rose from $61.48 to $96.21. These are totals for the benchmark runs, not prices for a single user request. Mem0 led in that configuration; another memory system, Honcho, scored highest in the paper's Claude Code tests.
Simulated workplaces, fixed conversation histories
The authors work at Mem0, and the results are their own evaluation. The conversations and apps are simulated, cover only three kinds of knowledge work, and do not test new history arriving between tasks. DolphinBench therefore offers a repeatable way to compare action, cost and delay under stated conditions, while performance in live, changing workplaces remains unproven.
