Visa has launched an AI shopping benchmark, and in its first round the AI models failed 42.8% of their test runs. The Visa Agentic Commerce Benchmark put 22 AI models from 12 labs through 15 shopping tasks, the company said in a statement on Thursday.
The tasks included buying a TV, a laptop and a personal care appliance. The customer did not give every detail up front. Each AI had to work out what they needed by talking to them. If nothing on offer fitted, the AI had to decide not to buy. To rule out lucky guesses, it also had to find enough information to justify its choice.
Visa calls it the first benchmark of its kind. To pass a run, an AI must find every requirement set in the task and make the right call to buy or not.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work.” Rubail Birwadker, global head of growth, Visa
Who passed
Across all tests, the models passed 57.2% of their runs. The best passed 82.2% of attempts, and the worst 28.9%.
Visa’s statement did not name the models. Agents from Google and Meta shared the top score of 82%, Jeff John Roberts reported for Fortune. A Claude agent came third with 76%, and an OpenAI agent sixth with 64%.
Asking the right questions helped. The five models that uncovered the most hidden buying factors passed 70.2% of the time. The five that uncovered the fewest passed 44.4%. The five most expensive models per run passed 67.1%, against 50.2% for the five cheapest.
Each model had three tries at every scenario, and all three tries passed only 34.2% of the time. In 54% of the failed runs, the AI still ended up making a purchase. It either picked a product that did not meet the customer’s needs, or bought before it had found all of them.
The TV test
In one scenario, a customer asks for help choosing a TV. They mention a brand a relative recommended and a membership with a third-party supplier, but leave out other details. The AI has to find the budget and the rest, such as format support, HDR10+ for family videos and gaming features.
Models that were better at finding the requirements passed more often, Visa said. But when the same models repeated a scenario they had already passed, they often failed.
Trust comes first
Visa’s Trust Index, published in September, found that 72% of US consumers had used an AI assistant. Only 23% trusted AI to make payments for them. Visa has already put Visa into ChatGPT and joined Mastercard and Ant International on ID checks for agents. A NIQ survey found that AI shopping passed 50% of US consumers in September.
Birwadker told Fortune that shopping by AI agent is still in its early days.
“We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.” Rubail Birwadker, Visa
Visa said the results do not mean AI agents are not ready for commerce, but that consistency and reliability remain problems. It plans to add scenarios and product categories, and to extend the tests to checkout, payment and ordering. It has shared the benchmark with leading model providers for feedback. Anyone can try the test on its Payments Frontier site.