BlogResearch
Reading Visa's AI shopping benchmark next to 904 real checkouts
Visa's benchmark and ours both lose about half their runs, and both lose them at the step where the agent has to work out exactly what the shopper wants.

Overview
Last week, on November 5,
Visa released the
Visa Agentic Commerce Benchmark. It put 22 models from 12 labs through 15 shopping scenarios, three times each, for 990 runs. 57.2% of the runs passed.
Visa's release leads with the other side of that number: almost half of test runs fail.
Since late September we have run a test of our own. An agent is signed in to a real store as a real member, given a request a person would type, and graded in code on whether it reaches a ready checkout with the correct total. Across 904 requests at
Amazon,
Walmart,
Target,
Chewy,
Uber Eats and
Instacart, 47% reached one.
We read
Visa's whitepaper and the results file behind its site. This post covers what the benchmark measures, where its findings match ours, and what a live store adds.
What
Visa measured
Visa built a simulated market whose contents are fixed in advance, so every model shops the same shelves. The assistant under test researches with tools for search, listings, reviews, store policies and price quotes. The customer is a second model holding a private list of requirements, most of them unstated until the assistant asks. A run passes when the customer ends up with an offer that meets every requirement, or correctly walks away when nothing fits, and every hidden requirement came out in the conversation first. The one scenario
Visa has made public is a TV market of 265 offers, 20 of which fit the customer.
The benchmark ends at the customer's decision. Payment and order placement are outside it, and
Visa lists checkout as a later stage of the work.
Two figures in the release are worth reading closely.
The first is this line: "About 1-in-2 (54%) of failed runs still ended in the AI making a purchase." In the benchmark, a purchase is the simulated customer accepting an offer. 229 of the 424 failed runs ended that way, with an offer that missed a requirement. No order was placed and no card was charged.
The second is that 72% of U.S. consumers have used an AI assistant. That comes from
Visa's Trust Index, a
Harris Poll of 2,065 U.S. consumers released in September, and the question is about using an AI assistant at all. In the same survey, 23% said they trust AI to handle payment transactions for them.
Where the two benchmarks agree
Runs are lost at the step where the agent has to work out exactly what the shopper wants.
In
Visa's results, the five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the fewest passed 44.4%. Of the 424 runs that failed, 343 ended in a wrong decision about the product. The other 81 hit a tool limit or returned an unusable reply.
In ours, 7 in 10 stops come right after the search results load. Opening the store and signing in cost almost nothing: 98% of requests were still going after sign-in. Once the store's results held exactly what was asked, 84% of requests reached a ready checkout, against 47% overall.
Visa also reports that no model clearly leads. Its top two,
Gemini 4 Argon and
Muse Wonder, each passed 82.2%, and the lowest passed 28.9%.
DeepSeek V4.1 Flash appears in both benchmarks. It passed 57.8% of
Visa's runs. In our lab, on the same 94 tasks, it reached a ready checkout 34% of the time and
GPT-5.6 Sol 22%, at a median model cost of under one cent a run.
What a live store adds
A live store puts three things in front of an agent that a simulated market leaves out: a sign-in, a long path to payment, and a price that changes on the way.
The sign-in comes first. Every run is conducted as though an AI is supporting a real signed-in member of that store. Customers have accounts because they bought there before and want to buy again. Agent sign-ins worked 49% of the time.
The path is long. At the median store the payment page is 13 screens in, and 29% of screens had a pop-up, banner, notice or check covering the page.
The price moves. At 10 of 25 stores the order total was higher than the first price shown, typically by about 27%, and shipping caused 7 of the 10.
When the path is clear, it is quick. A ready checkout on
Amazon takes about 40 seconds at the median.
Why a failing half is good news
72% of U.S. consumers have used an AI assistant, and about half of AI shopping runs fail in either benchmark. It is good that most agent purchases cannot check out yet. It shows how much room for growth there is in agentic commerce, and the stores that clear the path first take the sales.
We already see it. Shopping for the same two seats, our agent was let straight in at
SeatGeek and
StubHub and paused at sign-in for 16 hours at
Ticketmaster. In our lab, agents reached a ready checkout on 67% of
Amazon requests and 24% at the other five stores together. The store that is easiest for agents wins the sale, whatever its size, and that gives a smaller retailer a way to level the playing field.
What comes next
Visa plans to add more shopping scenarios and product categories, and to extend testing to checkout, payment and order placement.
Checkout Bench runs that stretch today on real stores, with more than 100 top U.S. retailers on its list and over 500,000 recorded network calls behind its findings.
If you run a store, we can send agents through your checkout and show you each place they stop. Request a private test of your store.
Sources
- Visa's release, November 5, 2026
- The benchmark's page, with per-model results
- Visa's whitepaper
- Visa's results file, from which we counted the 343 and the 81
- Visa Trust Index
- Every number about our own runs is on the Research page.