Before testing an open-weight model with customers, Nubank researchers used more than 16,000 synthetic conversations to screen 29 configurations of their support agent. In a live A/B test against the bank’s incumbent GPT-5.2 model, the selected Qwen3.5 model with a revised prompt increased the share of cases solved without human support by 8.82 percentage points; a customer-satisfaction measure showed no statistically significant change, according to a Sept. 24 arXiv preprint by Nubank and Guardrails AI researchers.
The simulator, Snowglobe, creates customer personas from different use cases and conversation styles, then has them talk to the support agent across multiple turns. When the agent calls a tool—a software function used to retrieve information or take an action—the simulator intercepts it and returns scenario-matched synthetic data. This lets the team test dialogue and tool use without calling production backends, but does not test the real services or data behind those tools.
The 29 configurations varied model family, reasoning effort and numerical format. Evaluations checked whether the agent gathered required information, handled card reissues correctly, retrieved account or card status, and completed the conversation. Qwen3.5-122B-A10B with reasoning enabled was selected as a starting point for prompt optimization, not because it led every measure: its status-retrieval failure rate was 35.2%, compared with 2.4% for the incumbent GPT-5.2 model. The authors say prompt changes reduced that gap before the live comparison.
The researchers also checked whether synthetic transcripts resembled real customer chats. In a blind sample of 100 conversations, seven domain experts correctly identified 70% of simulated chats and 84% of real chats. The authors say synthetic behavior still differs enough that simulated scores should not be treated as a substitute for customer testing.
In the live A/B test, the candidate replaced GPT-5.2 inside Nubank’s Card Management support agent. Self-service rate—the share of sessions resolved without a request for human support—increased by 8.82 percentage points (95% confidence interval: 7.95 to 9.69; 8,400 sessions). Transactional net promoter score, a post-interaction survey measure, changed by −1.21 points (95% confidence interval: −3.97 to 1.55; 2,300 responses), which was not statistically significant. The serving stack’s 95th-percentile latency—the response time below which 95% of observed responses fell—was 25% lower, a point estimate for which the paper reports no confidence interval.
Sources
- Original study: Screen Before You Serve
- Snowglobe documentation: simulation prompts
- Lead image: Figure 1 from the original study by its authors, licensed under CC BY 4.0; the diagram is not a production screenshot.