Nubank cuts AI agent update cycles from three weeks to under five days by testing on simulated customers first
Research shows that simulated customers can help Nubank test more scenarios, make decisions faster, and improve customer outcomes.
Nubank’s AI Research team has found that testing AI agents against synthetic customers in large-scale simulation, before they reach real customers, can speed up development and help teams choose better versions. “In the study, each new agent version took an average of 4.4 days with simulation, compared with 21.2 days in an earlier workflow without it. In a live A/B test, the agent built with simulation recorded a 4.9-percentage-point higher self-service rate and a 36.7-point higher transactional Net Promoter Score (tNPS), a measure of customer recommendation after an interaction, than a comparison agent.” – according to the press release.
Described in a new research paper, “Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale”, the approach uses a controlled testing environment in which synthetic customers with different profiles and needs hold multi-turn conversations with AI agents, so changes can be evaluated before they reach real customers.
The study focuses on two customer-support agents across the credit card lifecycle: Card Delivery, which handles delivery status and reissues, and Card Management, which covers address updates and customer-requested reissues. Simulation was used to guide changes to Card Management, while Card Delivery served as the comparison agent in the live A/B test.
“Every change to an AI agent is a bet on the customer experience. Simulation lets us place those bets on synthetic customers first, so only the changes that clearly win reach real people,” said Rohan Ramanath, General Manager of AI at Nubank.
Turning faster learning into customer impact
Simulation helped the team decide which changes to the Card Management agent were worth taking to real customers. In offline evaluation, one version of the Card Management agent reduced unnecessary handoffs to human support from 22.4% to 16.0% — a decrease of 6.4 percentage points compared with the incumbent version.
The subsequent live A/B test then measured the impact with real customers. Compared with Card Delivery, Card Management showed a self-service rate 4.9 percentage points higher and a tNPS 36.7 points higher. Because Card Management handles a broader set of tasks than Card Delivery, these gains reflect both the new agent and the simulation-guided process used to build it. Following the test, the Card Management agent became available to Nubank’s customer base in Brazil.
Self-service rate represents the share of sessions completed with the agent without a request for human support. On its own, it does not prove that every customer issue was successfully resolved. That is why the analysis also considers satisfaction metrics and specific quality evaluations.
A faster and broader way to evaluate AI agents
Without a simulation layer, testing new versions of an agent can depend heavily on manually written scenarios, limited samples, and experiments with real customers. Large-scale simulation allows teams to compare more combinations of prompts, models, and agent architectures in a controlled environment before deciding which changes should move forward.
The same approach helped Nubank evaluate open-weight models for customer support. The team screened 29 model configurations across more than 16,000 simulated conversations, then selected Qwen with reasoning enabled for further refinement. Simulation exposed a weakness that the team addressed through prompt optimization before live testing.
In a separate live A/B test against the incumbent Card Management model, the refined Qwen configuration increased self-service by 8.8 percentage points and reduced p95 response latency by 25%, with no statistically significant change in tNPS.
What the research opens up
The paper’s broader contribution is to show that evaluating AI agents requires more than checking whether they produce the right answer. It also suggests that a simulator’s job is to rank alternatives correctly, not to replicate human behavior perfectly, and that simulators must themselves be validated before teams rely on them. These ideas point to new research directions: standard tests for simulator reliability; improvement loops in which agents are revised, screened and refined with less manual effort; and evaluation of more proactive agents that guide customers through tasks.
The current approach stops at the systems an agent calls, so extending simulation to backend behavior and response times is a natural next step. Because every change is tied to a hypothesis and a success criterion set in advance, the workflow also creates a clear record of why each change was approved, which is valuable in regulated industries.
For Nubank, this approach connects applied AI research with a practical business challenge: how to develop more useful and reliable customer experiences as conversational AI becomes part of more interactions.
