Visa releases first-of-its-kind industry benchmark for AI shopping; almost half of test runs fail
- First-of-its-kind1 benchmark tests AI in realistic shopping scenarios—and consumers can try the test, too
- AI should uncover customer needs and make an informed purchase decision; models failed 42.8% of test runs
- Benchmark creates a roadmap for industry players to deliver better agentic shopping experiences
Visa (NYSE: V) launched Visa Agentic Commerce Benchmark, “a new benchmark for industry that puts leading AI models through realistic buying challenges—and is letting people take the test, too” – according to the press release.
Shopping is a difficult task for AI because consumers often do not clearly know what they want at the outset—for example, they may know they want a TV, but not all the required specifications. “Under Visa’s new benchmark, AI models are required to identify the needs of a consumer based on a conversation. If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.” – the company said.
The first round of tests involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a TV, laptop, or personal care appliance.
The Results
- Room for improvement: Across testing, selected AI models achieved an overall benchmark pass rate of about half (57.2%). The best performing models passed 82.2% of attempts, the worst 28.9%.
- The more you know: The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a 70.2% pass rate. The five that uncovered the least passed less than half (44.4%) of the time.
- Pricy models win, but do not guarantee success: The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five least expensive.
- Consistency is an issue: Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time.
- Buyer beware: About 1-in-2 (54%) of failed runs still ended in the AI making a purchase. Either the AI recommended a purchase that did not meet the customer’s identified requirements, or it failed to uncover all the customer’s requirements.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. “Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
Superintelligence Sweep
In one scenario, the customer asks for help choosing a new TV. They mention their relative has recommended a brand of television, and that they hold a membership with a third-party supplier—but leave other details unsaid.
Through conversation, the AI must uncover the customer’s budget and additional requirements—such as format compatibility, HDR10+® technology support for family videos, and gaming capabilities.
Models that were better at uncovering the customer’s requirements tended to pass more often. But when made to repeat a scenario they had already passed, the same models often failed—showing how difficult it is to deliver a consistent consumer experience.
Eliminating errors and improving consistency is crucial to building trust in agentic commerce, which is still relatively low today. Visa’s Trust Index, released in September 2026, found that while 72% of U.S. consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf. Visa Intelligent Commerce, Visa’s portfolio of products, programs and partnerships for agentic commerce, is designed to tackle this challenge.
Visa plans to add other shopping scenarios and product categories to Visa Agentic Commerce Benchmark, and will extend testing to checkout, payment, and order placement. Visa has shared the benchmark with leading model providers for feedback.
Visit Visa Payments Frontier to attempt Visa Agentic Commerce Benchmark yourself, watch replays of AI attempts, and dig deeper into the research with the Visa Agentic Commerce Benchmark whitepaper.
FAQs
Is Visa Agentic Commerce Benchmark part of Visa Intelligent Commerce?
Not directly. Visa Intelligent Commerce is a portfolio of products, programs and partnerships that help bring trust to agentic commerce. The benchmark is complementary research designed to benefit the wider industry.
Is Visa saying AI agents aren’t ready for commerce?
No. The results show promising capabilities, with the highest-scoring models each passing 82.2% of attempts. But consistency and reliability remain challenges.
Is Visa Agentic Commerce Benchmark publicly available?
Yes. Anyone can explore the research and interactive experience on Visa Payments Frontier, including consumers, researchers, developers, merchants, clients and AI companies.
Why is Visa benchmarking AI models?
To help make agentic commerce more reliable. The benchmark gives the industry a consistent way to understand where AI shopping works today and what still needs to improve.
___
1 Visa Agentic Commerce Benchmark is the first agentic commerce benchmark to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass.
