AI Shopping Agent Checkout Benchmark

For retailersWe send shopping agents to your site and show you where they get stuck.

Can an AI agent buy from your store?

People are starting to hand their shopping to AI assistants. When an assistant can't find the right size, gets lost in a pop-up or stalls at checkout, the sale just doesn't happen, and nobody tells you. We find those places before your customers do.

Request a private test of your store Where agents get stuck

Methodology

The Checkout Bench benchmark measures whether an AI shopping agent can take a person's request and reach a ready checkout at a real store. Each run happens in a fresh cloud browser inside real, signed-in store accounts, using the kinds of requests people actually type.

Runs are graded in code from the run's record, and no model grades another model. A run passes only when the agent picked the right item in the right quantity, stayed within any budget the person gave, and asked a question when the request was really unclear. An order is placed only after the person says yes to the checked total, and a purchase counts once the store confirms the order.

The stores covered are Amazon, Uber Eats, Instacart, Walmart, Target and Chewy. The assistant models in these results are GPT-5.6 Sol, DeepSeek V4.1 Flash, GPT-5.6 Luna, Claude Opus 4.8, Grok 4.7 and Kimi K3.

This page was last updated on November 6, 2026.

Real runs so far, on signed-in accounts at

Amazon

Walmart

Target

Instacart

Uber Eats

Chewy

Report · Updated Nov 3, 2026

Where AI shopping agents get stuck, and what exists to test it

What real quote runs on signed-in store accounts show, what a retailer gets back from a test, and what the system does today.

Data: 458 quote-lab runs · 30 reviewed Test flight runs
Read it as: Markdown · llms.txt · the runs

35%
of 527 real checkout requests ended in a ready checkout. The biggest drop is from search results to checkout read: 518 to 231. Figure 1
11 of 30
reviewed runs went wrong on the assistant's side, 5 on Checkout Bench's, and 7 quoted correctly. Figure 2
40
stores and sites can be connected today, and 7 return a full cart and total. Section 04

Where agents get stuck

An AI agent goes through the same five steps a person does. Each one is a place a sale can quietly fail.

Who runs, where they go

Who runs, where they go, what comes back.

Model → agent → store → ready checkout Real · counts per store

SoGPT-5.6 Sol

DSDeepSeek V4.1 Flash

LuGPT-5.6 Luna

OpClaude Opus 4.8

Lab models

Checkout Bench agent

Signed-in cloud browser

Amazon

Uber Eats

Instacart

Walmart

Chewy

Target

Ready Checkouts

Amazon

101of 188

Median quoted total: $11.00
Median time: 34.6 s

Four models feed one agent, which fans out to the six stores and returns a ready checkout. The quote card cycles through each store's real median time and total.

1 · Search

What goes wrong. The agent searches the way a person talks and gets nothing useful back.

What we check. Do ordinary requests land on the right product?

2 · Product options

What goes wrong. Size, color or pack count is chosen in a way agents can't read, so they stop or pick the wrong one.

What we check. Can an agent pick the variant the person asked for, even when the store calls “green” “Lime”?

3 · Sign-in

What goes wrong. Logins, pop-ups and account checks block the agent before it reaches the cart.

What we check. Does a signed-in session get through without a person stepping in?

4 · Cart

What goes wrong. Quantities, bundles and add-ons end up different from what was asked.

What we check. Does the cart hold exactly the right item and amount?

5 · Checkout

What goes wrong. Fees, delivery choices or a final review screen the agent can't get past.

What we check. Does it reach the final review screen with the right total?

From checkout request to a ready checkout

How far 527 real quotes got through the same steps a shopper takes. Tap a stage for its numbers.

Anonymized

Checkout requested100.0%Browser ready99.4%Signed in98.3%Search results98.3%Checkout read43.8%Quote checked37.2%Ready Checkout35.3%

527 → 186 quotesOverall conversion 35.3%

Each stage counts the quotes whose record reached it. “Product chosen” is left out because one quote can choose more than one product. Sign-in holds up well; most quotes stop between search and checkout, at the product choice, and that is usually Checkout Bench stopping to ask, not an error.

Source: Checkout Bench Research527 quotes

What you get back

A score for each step, every place an agent got stuck, and whose problem it was: your site, the agent or our engine.

The report

A score for each step, from search to checkout
A replay of every place an agent got stuck (coming soon)
A fix list, ordered by how many runs each problem blocked
Every result split into “your site” and “the agent”, so you only fix what's yours

Re-checks

After you ship fixes we run the same requests again, and on a schedule after that. If a site change starts breaking agents, you hear about it with the step that broke.

yourstore.example: agent readiness

Out of 100. Agents most often get stuck choosing size and color, and at the final checkout step.

88%

Product options

41%

Sign-in

72%

Cart

80%

Checkout

35%

Whose problem it was, store by store

Every result split into the store, the assistant and our own engine. Tap a store to see its mix.

Anonymized

Amazon · 12 runs100%

Amazonn=12Walmartn=4Targetn=2Instacartn=4Uber Eatsn=6Chewyn=2

Share of each store's Test flight runs by the reviewer's verdict: 15 real prompts, each run once with DeepSeek V4.1 Flash and GPT-5.6 Luna on the real store sites. One sample per case and model, labeled by a reviewer reading each record, so read it as where the problems were, not as a ranking.

Source: Checkout Bench Research30 runs · Oct 25, 2026 PDT

A private test environment built on your store

We set the test up on your own catalog and checkout, with the assistants your customers actually use.

How it works

We work with your team and your own product data: your catalog, your variants and your real checkout flow.
We build a test setup that is yours alone, with the requests your customers actually make.
We run the assistants your customers are starting to use, such as ChatGPT, Gemini and Perplexity, through it and show exactly where each one gets stuck.
Nothing is ever bought, and your data stays in your environment.

Request a private test of your store

Tell us your store, the flows to test and the assistants to use. We set up the test and send the report.

Request a private test of your store

Why trust the numbers

The same runs power our public benchmark. Every run happens in a fresh cloud browser on a real store and is graded in code, not by another AI.

What exists today · Test suite

For developers

The engine that drives these runs is also available as an API for teams building assistants that shop. Details are further down the page.

The API · Docs

What exists today

Anonymized

From 458 Checkout Bench lab runs and the store list. More on the test suite.

stores and sites a person can connect an account to today: 12 built in, 28 set up from their web address

stores where a full cart and total come back today: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE

fresh cloud browser per run, thrown away afterwards

384

trace events in one real run: DeepSeek V4.1 Flash asked Amazon for a DJI mic; Checkout Bench blocked all five quotes as product mismatch

Quotes by store

Ready versus blocked quotes at leading retailers.

Anonymized

Amazon101/188
Uber Eats34/131
Instacart23/102
Walmart13/56
Chewy10/27
Target5/24

Ready CheckoutBlockedStill preparing

Source: Checkout Bench Research527 quotes

Why quotes were blocked

Top ten blocker codes. Most are the agent pausing to ask the person, not errors.

Anonymized

product_choice_requires_user140
product_mismatch19
checkout_link_missing12
item_dialog_missing12
below_store_minimum12
store_not_found12
item_choices_required11
quote_failed10
over_budget10
checkout_not_ready9

Source: Checkout Bench Research336 blocked quotes

Store coverage

Which stores connect, and which also quote and buy.

Anonymized

AmazonConnect · quote · buy
Uber EatsConnect · quote · buy
InstacartConnect · quote · buy
WalmartConnect · quote · buy
TargetConnect · quote · buy
ChewyConnect · quote · buy
DoorDashConnect only
ResyConnect only
DETOUREConnect · quote
DeltaConnect only
AppleConnect only
Urban OutfittersConnect only

Quote (7)Account connect only (5)

Plus 28 sites we set up from their web address, connect only (Skyscanner also offers flight search and price alerts).

Source: Checkout Bench Research

Lab runs per day

Quote tests started in the Checkout Bench lab, as a calendar (Monday at the top).

Anonymized

SepOctMonWedFriSun

Runs: 458
Busiest day: Oct 1 · 264
Days with runs: 7 of 30

lessmore

Tap a day for its count; tap a second day to total the span between them, a third tap clears. Days outside the record window are left blank.

Source: Checkout Bench Research

Same system, five businesses

Each card is a complete pitch for the same system: who buys, what they get, how it works and a rough price shape.

Benchmark

How well do AI agents actually shop?

We hand models the requests real people type, and let them work inside real, signed-in store accounts at Amazon, Walmart, Target and four more. Code checks the final cart. We publish what passed and what didn't.

Pass rate, all 6 stores

Model A

62%

Model B

55%

Model C

47%

Model D

33%

Model E

25%

Each model runs the same 240 requests on real store accounts.

Who buys it, what they get, how it works, pricing

Who buys it

Model labs and agent companies that want an outside number for shopping tasks, and the people choosing between models.

What they get

A public leaderboard by model and by store
Private runs on models that aren't released yet
The full step-by-step trace behind every pass and fail

How it works

A request written the way people write: “can you get me the dji mic mini thing from amazon”
The agent works in a fresh cloud browser on the real store, all the way through checkout
Code checks the item, the quantity and the price cap

Pricing idea

Leaderboard is free to read. Private evaluation rounds are priced per model, per round (not set yet).

See the example leaderboard

RL training environment

A practice store for training shopping agents

Recorded copies of real store pages that reset instantly, a bank of messy requests with the right answers written down, and graders written in code. For labs training agents that buy things.

Task T-001

Shopper said

“can you get me the dji mic mini thing from amazon”

Should ask firstAmazon

store: amazon (recorded copy, resets per episode)
accept: any DJI Mic Mini kit
must_ask: true: several kits match
cap: none
grader: item · qty · cap · no_purchase · asked

env = CheckoutGym(task="T-001")
obs = env.reset()          # same start every time
obs, done = env.step(action)
score = env.grade()        # code, not a model

Who buys it, what they get, how it works, pricing

Who buys it

Research and post-training teams at model labs.

What they get

A container with a reset / step interface that runs many copies in parallel
Labeled tasks: each one lists the products that count as correct, or says the right move is to ask
Graders in code, no model judging another model
A held-out set of live runs to check the copy still matches the real store

How it works

Record real sessions at each store, with personal details removed
Replay them as a store that resets in milliseconds and behaves like the original
Score every episode in code and keep the live exam separate from training

Pricing idea

Licensed as task packs. Comparable deals we have seen land around $200–2,000 per task depending on difficulty and exclusivity. A range, not a quote.

See a task and its grader

For retailers

Can an AI agent buy from your store?

We send shopping agents to your site with ordinary requests and show you where they get stuck: search, product options, sign-in, cart or checkout.

yourstore.example: agent readiness

Out of 100. Agents most often get stuck choosing size and color, and at the final checkout step.

88%

Product options

41%

Sign-in

72%

Cart

80%

Checkout

35%

Who buys it, what they get, how it works, pricing

Who buys it

E-commerce and growth teams at retailers and brands, and the platforms that host their stores.

What they get

A score for each step from search to checkout
A recording of every place an agent failed
A fix list ordered by how many runs each problem blocked

How it works

Pick 20–50 requests your customers really make
Several different agents try each one in fresh browsers
Read the report, fix things, run it again

Pricing idea

A flat fee per audit, with an optional monthly re-check.

See recent real runs

Monitoring

Know when a store change breaks shopping agents

We run a fixed set of shopping tasks against each store on a schedule. When a site change makes them fail, you get an alert with the step that broke and a recording of it.

Alerts

Target · add to cart started failing at 3:12 AM PDT
9 of 10 tasks passed yesterday, 2 of 10 today. The size picker moved into a pop-up and agents no longer find it.

Step that broke

Before: size buttons on the page · After: “Choose options” opens a sheet

Other stores

Amazon, Walmart, Instacart, Uber Eats, Chewy: no change

Who buys it, what they get, how it works, pricing

Who buys it

Teams that ship shopping agents or browser automation, and retailers who want agents to keep working on their site.

What they get

Scheduled runs per store
Alerts by email, Slack or webhook
A before-and-after comparison of the step that broke

How it works

Choose the stores and tasks you care about
We run them every few hours in fresh browsers
When something that passed starts failing, you get the step, the screenshot and the difference

Pricing idea

Monthly, per store watched.

See a blocked real run

API

One API to sign in, fill a cart and check out

Your agent sends what the person asked for and which connected store to use. We sign in, find the product, build the cart and send back the total. The order is placed when the person says yes.

Request and response

POST /v1/carts
{ "account": "acct_amazon_123",
  "request": "can you get me the dji mic mini thing" }

200 OK
{ "status": "needs_choice",
  "options": [
    { "title": "DJI Mic Mini, 2 TX + 1 RX", "price": "…" },
    { "title": "DJI Mic Mini, phone kit",   "price": "…" } ],
  "ordered": false,
  "trace": "trc_…" }

Who buys it, what they get, how it works, pricing

Who buys it

Developers building assistants that need to buy things for their users.

What they get

Account connection at 40 stores and sites
Cart and total at 7: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE
Orders placed only after the person confirms
A full trace of every request

How it works

The person connects a store account once
Your agent calls the API with their request
You get the cart and total back, and place the order only after they confirm

Pricing idea

Paid per cart total returned and per order placed (rates not set yet).

See a real run timeline

What the directions would measure

Layout previews for three of the directions. Each is built on a real piece of the system (the lab's models, the Kilowatt test store's cases, the grader's price rule).

A · Benchmark

Model rank by store

Option A: where each lab model would rank on ready checkouts at each store. Tap a line or a name to follow one model.

Rank at each store
Model	Amazon	Walmart	Target	Instacart	Uber Eats	Chewy
GPT-5.6 Luna	2	1	2	3	1	2
DeepSeek V4.1 Flash	1	2	3	1	2	1
Grok 4.7	3	4	1	2	4	3
Gemini 3.8 Flash	4	3	4	4	3	4

The four models are the ones the lab runs today.

Source: Checkout Bench Research1 = best

B · Training environment

Price check ladder

Option B: quoted item subtotals stacked around a case's reference price, the way the grader sees them.

$349.00Allowance ±$6.98 · 2.0%

Under 67.6%Visible quotesOver 32.4%

Tap a level to see how many quotes sit between it and the reference.

The case and the rule are real: KW-color-01 on the Kilowatt test store expects a $349.00 item subtotal, and the grader passes a quote within max(2% of that, 50¢) = ±$6.98, before tax and shipping. 27 of 37 would pass the price check.

Source: Checkout Bench ResearchKilowatt test store

D · Monitoring

Quoted totals through a day of checks

Option D: the same task run on a schedule, each column one check. Tap a cell for its count.

Agent quotes per total · 8 runs per check · line = median · $30.00 cap08

The case is real: KW-cable-01 asks for two 6 ft braided USB-C cables under $30.00 total, and the reference checkout is $28.12 with free shipping and test tax. When a $2.99 shipping fee lands at 15:00, every quote goes over the cap, and that is the moment an alert would fire.

Source: Checkout Bench Research16 checks · every 90 min

Plain answers

Is this a real product?

Yes. Checkout Bench runs AI shopping agents through real, signed-in store checkouts and shows where they get stuck. Retailers can request a private test of their store, the test suite at /lab lets signed-in users run assistants against real quotes, and agent developers can use Checkout Bench's API, documented under docs.

What exists today?

Account connection at 40 stores and sites (12 built in, 28 set up from their web address); a cart and total at 7 (Amazon, Uber Eats, Instacart, Walmart, Target, Chewy and DETOURE), and at the first six the order itself once the person says yes; a fresh cloud browser for every run; a full trace of each run; and the test suite at /lab. One real run, DeepSeek V4.1 Flash asked for a DJI mic on Amazon, recorded 384 trace events, and all five quotes were blocked as product mismatch (the checkout title was longer than the product page's).

How is a run graded?

In code, from the run's record: right item, right quantity, within any budget the person gave, and a question when the request was really unclear. A purchase counts only once the store confirms the order. No model grades another model.

When does an order get placed?

When a person says yes. A purchase needs the person's explicit yes to that quote's total, and it counts only once the store confirms the order.

Are the numbers on these pages real?

The facts, charts and recorded runs are real and marked Real.

Recent runs

Anonymized7 runs

The most recent real runs, newest first: runs from Checkout Bench run records, plus one direct Best Buy tool run with no model. Each one is a real quote or read on a real store. Steps are condensed from the recorded evidence and scrubbed for this public page.

Store	The shopper said	Result	Price	Model	Steps	When
Amazon	“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”	Quoted`Reached order review`	$5.38	typesafe-ai/jev	4	Oct 26, 4:12 PM
Best Buy	“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”	Product page only`Found product, no cart or checkout`	$14.95 before tax	No assistant (direct tool calls)	4	Oct 26, 4:45 PM
Amazon	“Re-run of the last DJI request from the run below, on the exact kit the person had chosen”	Quoted`Ready Checkout`	$87.10	No assistant (direct engine check)	6	Oct 25, 9:12 PM
Amazon	“can you get me the dji mic mini thing from amazon”	Blocked`product_mismatch ×5`	$35.00 and $79.00 listings	DeepSeek V4.1 Flash	7	Oct 25, 8:21 PM
Instacart	“Regular Coca-Cola (exact message not recorded)”	Needs the shopper`product_choice_requires_user`	No total	DeepSeek V4.1 Flash	5	Oct 25
Amazon	“Crest toothpaste (exact message not recorded)”	Quoted`Ready Checkout`	$4.33	DeepSeek V4.1 Flash	6	Oct 25, 7:55 PM
Uber Eats	“Status of an earlier Uber Eats order (read-only continuation; exact message not recorded)”	Status read`Delivered (read)`	n/a (read only)	DeepSeek V4.1 Flash	5	Oct 25, 7:57 PM

Source: Checkout Bench Research. The raw per-event trace files are kept in the lab's private run history and are not published; IDs, accounts, addresses and card details are removed.

Charts

Every chart here is built from real records. No run IDs, prompts, accounts or addresses are shown.

Anonymized

Lab runs per day

Quote tests started in the Checkout Bench lab, by day.

Anonymized

Sep 2676Sep 2739Sep 2812Sep 2925Oct 1264Oct 213Oct 2529

Source: Checkout Bench Research

Run outcomes

Final verdict for each assistant run.

Anonymized

Quote ready16828%
Quote blocked24040%
Asked the person9516%
No quote6210%
Other315%

Source: Checkout Bench Research596 assistant runs

Quotes by store

Ready versus blocked quotes at leading retailers.

Anonymized

Amazon101/188
Uber Eats34/131
Instacart23/102
Walmart13/56
Chewy10/27
Target5/24

Ready CheckoutBlockedStill preparing

Source: Checkout Bench Research527 quotes

Why quotes were blocked

Top ten blocker codes. Most are the agent pausing to ask the person, not errors.

Anonymized

product_choice_requires_user140
product_mismatch19
checkout_link_missing12
item_dialog_missing12
below_store_minimum12
store_not_found12
item_choices_required11
quote_failed10
over_budget10
checkout_not_ready9

Source: Checkout Bench Research336 blocked quotes

Time to a quote

Seconds from checkout request to a finished quote, by store.

Anonymized

Walmart n=5434.3s · 74.5s
Amazon n=18834.6s · 60.7s
Instacart n=10139.3s · 49.6s
Uber Eats n=12946.2s · 117.8s
Target n=2356.8s · 115.7s
Chewy n=2763.1s · 90.3s

MedianUp to p90

Source: Checkout Bench Research522 finished quotes

Monitor the AI agent trace down to each 0.01s.

Median seconds into a quote when each step was reached.

Anonymized

8.2sBrowser ready
16.2sSigned in
23.3sSearch results
26.4sProduct chosen
39.7sCheckout read
43.9sQuote checked

0s10s20s30s40s50s

Medians over the quotes that reached each stage (n from 196 to 548), so later stages are measured on fewer quotes.

Source: Checkout Bench ResearchStage timings in quote records

Trace events per run

How many events one assistant run records (requests, tool calls, model turns, VM steps).

Anonymized

0242513950597520010064125381502717512200922552506300+13

Source: Checkout Bench ResearchMedian 82 · p90 158 · max 589

Assistant models and Cost Analysis

Runs, trace size, time and model cost by assistant model.

Anonymized

Assistant model	Runs	Median trace events	Median time	Median model cost per run
GPT-5.6 Sol	381	77	47.2s	$0.0639
DeepSeek V4.1 Flash	129	95	64s	$0.0068
GPT-5.6 Luna	51	105	78.9s	$0.0047
Claude Opus 4.8	10	85.5	64.4s	$0.0924
Grok 4.7	10	82	56s	$0.0116
Kimi K3	9	84	58.4s	$0.0295

Models with 8 or more runs. Model cost is the provider usage reported per run; it excludes the cloud browser.

Source: Checkout Bench Research

Experiment with Quote Totals and Account Budgets.

Ready Checkout totals at each store, cents included (tax, fees and tip as checkout showed).

Anonymized

Amazon n=101$11.00
Uber Eats n=34$26.80
Instacart n=23$26.80
Walmart n=13$13.10
Chewy n=10$9.30
Target n=5$40.50

Median totalLowest to highest Ready Checkout

Source: Checkout Bench Research186 Ready Checkouts

Listing price vs quoted total

Two documented runs.

Anonymized

DJI Mic Mini (2 TX + 1 RX) Oct 25, 9:12 PM
$79.00 listing+$8.10
$87.10 quoted · +10.3%
Crest Pro-Health toothpaste 4.3 oz Oct 25, ~7:59 PM
$3.93 listing+$0.40
$4.33 quoted · +10.2%

The listing price is what the product page showed; the quoted total is what checkout showed, including tax. In both runs the difference is tax only.

Source: Checkout Bench Research

Store coverage

Which stores connect, and which also quote and buy.

Anonymized

AmazonConnect · quote · buy
Uber EatsConnect · quote · buy
InstacartConnect · quote · buy
WalmartConnect · quote · buy
TargetConnect · quote · buy
ChewyConnect · quote · buy
DoorDashConnect only
ResyConnect only
DETOUREConnect · quote
DeltaConnect only
AppleConnect only
Urban OutfittersConnect only

Quote (7)Account connect only (5)

Plus 28 sites we set up from their web address, connect only (Skyscanner also offers flight search and price alerts).

Source: Checkout Bench Research

Test suites passing

Tests passing in each suite at the last full check.

Anonymized

Cloud436/436
Lab170/170
Service140/140
Onboarding18/18

Source: Checkout Bench Research

Activity

Entries per day, shown as a heat calendar.

Anonymized

29Sep 17
18Sep 18
14Sep 22
38Sep 23
3Sep 25
28Sep 26
42Sep 27
28Sep 28
13Sep 29
24Oct 2
8Oct 25

245 entries on 11 days with any entry. Days with none are left out.

Source: Checkout Bench Research

Recent runs

Amazon

QuotedReached order reviewReal · Oct 26, 2026, 4:12 PM PDT (ran 41.2 s)

Signed-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. Score 1.0.

Shopper said

“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”

Timelinecondensed from the recordAnonymized

start0 s
Signed-in cloud Chrome session
searchno time in record
Right item found: Amazon Basics USB-C to USB-C cable, 6 ft
checkno time in record
Reached order review: $5.38 total vs $15 cap
done41.2 s
Order review read with the right item and total

A time is shown only where the record has one (seconds are from the start of that run).

Result

QuotedReached order review

Price$5.38

Signed-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. Score 1.0.

StoreAmazonModeltypesafe-ai/jevProviderNot recordedEngineCheckout Bench v3 Amazon quote driver (signed-in cloud Chrome)

Recent runs

Best Buy

Product page onlyFound product, no cart or checkoutReal · Oct 26, 2026, 4:45 PM PDT (ran 35.9 s)

This run ended on the product page because the tool has no cart step, unlike the Amazon run. It was signed out and used no model.

Signed-out run that found a 6 ft USB-C to USB-C cable listed at $14.95 before tax, under the $15 cap, with free shipping by Oct 30. The Pomerado tool has no cart step, so no cart or order total was seen. The checks below cover only what a product page can show.

Shopper said

“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”

Timelinecondensed from the recordAnonymized

start0 s
Pomerado Best Buy tool, signed out, shipping ZIP set
search20.0 s
20 listings; right item found: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, $14.95
read35.9 s
Product details: SKU 10555185, $14.95, Shipping FREE Fri, Oct 30; pickup not listed
done35.9 s
Product quote read; this tool has no cart step

A time is shown only where the record has one (seconds are from the start of that run).

Result

Product page onlyFound product, no cart or checkout

Price$14.95 before tax

This run ended on the product page because the tool has no cart step, unlike the Amazon run. It was signed out and used no model.

StoreBest BuyModelNo assistant (direct tool calls)Providern/aEnginePomerado Best Buy tool (search + product details), called directly

Recent runs

Amazon

QuotedReady CheckoutReal · Oct 25, 2026, 9:12 PM PDT

The same DJI Mic Mini (2 TX + 1 RX) that the live engine had blocked as product_mismatch, re-quoted with the title-alias fix.

Shopper said

“Re-run of the last DJI request from the run below, on the exact kit the person had chosen”

Timelinecondensed from the recordAnonymized

start0 s
Fresh cloud VM; the lab's managed Amazon session restored
readno time in record
Product page read: full document title extends the short visible heading; page address, canonical address and Buy/Add form agree on the chosen kit, so one title alias is recorded
cartno time in record
Exact chosen kit, quantity 1
checkno time in record
Checkout title matched the alias; quantity, seller, charges, requested address and requested card all matched
quote38.8 s
Quote published: DJI Mic Mini (2 TX + 1 RX) $79.00 + tax $8.10 = $87.10
done49.1 s
Session saved, VM closed

A time is shown only where the record has one (seconds are from the start of that run).

Result

QuotedReady Checkout

Price$87.10

The same DJI Mic Mini (2 TX + 1 RX) that the live engine had blocked as product_mismatch, re-quoted with the title-alias fix.

StoreAmazonModelNo assistant (direct engine check)Providern/aEngineCandidate engine, before release

Recent runs

Amazon

Blockedproduct_mismatch ×5Real · Oct 25, 2026, 8:21 PM PDT (ran 27 min 7 s; traced 9:03 PM PDT)

384 trace events. The assistant asked Checkout Bench for quotes on two DJI Mic Mini kits. All five quotes were blocked as product_mismatch: the checkout showed a longer product title than the product page, and no product ID at checkout.

Shopper said

“can you get me the dji mic mini thing from amazon”

Timelinecondensed from the recordAnonymized

requestno time in record
Person: “can you get me the dji mic mini thing from amazon”
listingno time in record
Kit seen: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone, $35.00
listingno time in record
Kit seen: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio, $79.00
checkno time in record
Checkout (2 TX + 1 RX): same wording plus extra descriptive text, 1 unit, $79 + $8.10 tax; no product ID readable
blockno time in record
Five quotes across the two kits: product_mismatch (exact-title rule)
cleanupno time in record
All five cart restorations recorded
replyno time in record
Assistant explained it as a same-product-ID issue and a transient error worth retrying; the record supports neither

A time is shown only where the record has one (seconds are from the start of that run).

Result

Blockedproduct_mismatch ×5

Price$35.00 and $79.00 listings

StoreAmazonModelDeepSeek V4.1 FlashProviderGatewayEngineLive engine

Recent runs

Instacart

Needs the shopperproduct_choice_requires_userReal · Oct 25, 2026 (start time not recorded, before 8:04 PM PDT)

Reached real sign-in and search, then stopped and asked the person to choose: the observed listing was a 24-pack and Checkout Bench could not tell it was the requested regular variant. No cart add, no purchase.

Shopper said

“Regular Coca-Cola (exact message not recorded)”

Timelinecondensed from the recordAnonymized

requestno time in record
Request included the saved address and default card
sessionno time in record
Real Instacart sign-in restored
searchno time in record
Search returned “Coca-Cola Soda (24 x 12 fl oz)”
blockno time in record
Quote blocked: product_choice_requires_user (fit to the requested regular variant unclear)
replyno time in record
Assistant said the title did not confirm the amount, and read preferences.mode:none as “no saved address or card”; both went beyond the evidence

A time is shown only where the record has one (seconds are from the start of that run).

Result

Needs the shopperproduct_choice_requires_user

PriceNo total

StoreInstacartModelDeepSeek V4.1 FlashProviderGatewayEngineLive engine

Recent runs

Amazon

QuotedReady CheckoutReal · Oct 25, 2026, 7:55 PM PDT (ran 1 min 9 s)

Live quote run through the lab's new CLI/API. Complete quote with the account's selected address and card and no warnings.

Shopper said

“Crest toothpaste (exact message not recorded)”

Timelinecondensed from the recordAnonymized

startno time in record
Test started from the lab CLI/API: live, quote only
quoteno time in record
Checkout read: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
checkno time in record
Address and card: the account's selected ones, as requested
quoteno time in record
$3.93 item + $0.00 shipping + $0.40 tax = $4.33
replyno time in record
Assistant reported the card the checkout actually showed instead of switching to its remembered preference
cleanupno time in record
cleanup: done, no warnings; assistant VM stopped

A time is shown only where the record has one (seconds are from the start of that run).

Result

QuotedReady Checkout

Price$4.33

Live quote run through the lab's new CLI/API. Complete quote with the account's selected address and card and no warnings.

StoreAmazonModelDeepSeek V4.1 FlashProviderGatewayEngineLive engine

Recent runs

Uber Eats

Status readDelivered (read)Real · Oct 25, 2026, 7:57 PM PDT (ran 47 s)

Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.

Shopper said

“Status of an earlier Uber Eats order (read-only continuation; exact message not recorded)”

Timelinecondensed from the recordAnonymized

startno time in record
Read-only continuation of an earlier purchase
statusno time in record
One order-status request sent to Checkout Bench
read7:58:09 PM
Order status: delivered (read from Uber's concluded-order view; scope: delivery status; receipt not re-checked)
replyno time in record
Assistant kept the Oct 2 receipt and ETA separate from the fresh delivery reading and reported no current ETA
doneno time in record
Assistant VM stopped; zero quote or purchase requests

A time is shown only where the record has one (seconds are from the start of that run).

Result

Status readDelivered (read)

Pricen/a (read only)

Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.

StoreUber EatsModelDeepSeek V4.1 FlashProviderGatewayEngineLive engine

Test flight

Anonymized15 cases × 2 models = 30 runs

One exact-prompt campaign run on Oct 25, 2026 against real store accounts. Each case was run once with DeepSeek V4.1 Flash and once with GPT-5.6 Luna. One sample per case, shared real accounts, not a model ranking.

Results updatedOct 25, 2026, 10:44 PM PDT

30 of 30runs finished and reviewed

28natural assistant finishes

2stopped by the operator (retry loops)

11reached a ready checkout

0purchase requests or purchases

43accepted quote operations, all settled

Case by modelclick a cell for the runReviewer label, not graded by code

Each cell groups the reviewer's outcome label into one of five plain buckets. The reviewer's own label is shown next to it on the run page. "Quoted correctly" counts quotes the reviewer did not fault; the record shows 11 runs reaching a ready checkout, some with caveats the reviewer flagged.

Quoted correctly7Asked a fair question5Checkout Bench backend bug5Assistant mistake11Stopped by operator2

Case

DeepSeek V4.1 Flash

GPT-5.6 Luna

A1Amazonugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1

Assistant mistake$4.33 proposal; usual tube unverified1 min 27.3 s Asked a fair questionAsked which usual tube54.0 s

A2Amazonneed that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one

Quoted correctly$38.59 mobile kit; iPhone port unconfirmed2 min 9.6 s Quoted correctly$38.59 mobile kit; iPhone port unconfirmed1 min 6.9 s

A3AmazonHi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing

Checkout Bench backend bugMatching 8-count pack refused1 min 5.3 s Checkout Bench backend bugMatching 8-count pack refused57.5 s

A4Amazonmy charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls

Quoted correctly$5.38; hard cap kept1 min 1.2 s Assistant mistake$5.38; preferred card omitted1 min 2.0 s

A5Amazonadd crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together

Assistant mistake$16.42; unsupported compatibility claim1 min 41.4 s Assistant mistake$20.61; handle fit unconfirmed1 min 21.7 s

A6Amazonok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50

Assistant mistake$88.85; soft target made hard1 min 37.8 s Checkout Bench backend bug$29.97 item; longer-title block1 min 10.9 s

U1Uber Eatsyo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]

Asked a fair questionAsked which saved address53.7 s Assistant mistakeSearched an unverified default address31.5 s

U2Uber Eatscan u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it

Stopped by operatorOperator stopped after 5 attempts4 min 43.4 s Checkout Bench backend bugDifferent options unsupported43.0 s

U3Uber Eatspizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one

Stopped by operatorOperator stopped after 7 attempts5 min 2.9 s Assistant mistakeMenu block; unobserved alternative suggested46.9 s

I1InstacartHello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you

Quoted correctly$26.92; store discovery worked1 min 52.1 s Assistant mistakeAsked for a store without searching9.1 s

I2Instacartrunning low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees

Checkout Bench backend bugSame-store items; 12 vs 4 egg-count conflict1 min 55.0 s Assistant mistakeAsked for a store without searching10.7 s

W1Walmartcan you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway

Quoted correctly$5.17 pickup; bag fee pending1 min 1.6 s Quoted correctly$5.17 pickup; bag fee pending1 min 7.5 s

W2Walmartget paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup

Quoted correctly$3.26 pickup; 2 double rolls1 min 0.6 s Asked a fair questionAsked among observed pack choices46.9 s

T1Targettarget run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff

Assistant mistake42-count unverified; shipping minimum unknown1 min 38.8 s Asked a fair question42-count unverified; shipping minimum unknown59.5 s

C1Chewydog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that

Assistant mistake30 lb bag unresolved; buy-once untested2 min 49.5 s Asked a fair question30 lb bag unresolved; buy-once untested1 min 9.6 s

How this was run

Prompts and expected outcomes were frozen before any run; all 30 prompt hashes match.
Each case shows the real prompt, scrubbed: tip amounts, saved-address labels, addresses, store locations, card brands and names are removed and marked inline.
Runs went one after another on shared real store accounts. One sample per case and model: not a model ranking or an independent-account benchmark.
Outcome labels and notes were written by a reviewer reading each run's record. They are not graded by code.
Prices and availability are what the run saw at the time, not current offers.
All assistant VMs stopped; every run with cart activity has cart-restore steps; zero unresolved quotes.

Source: Checkout Bench Research. Run IDs, replay links, store locations, saved-address labels, card brands and tip amounts are left out; removals are marked inline in the prompts. Catalog product numbers (ASINs, Instacart and Walmart IDs) are published.

Test flight

A1 · Amazon

DeepSeek V4.1 Flash

Assistant mistakepartialAnonymized

$4.33 proposal; usual tube unverified

Real prompt, scrubbedfrozen before the runs; hash checked for both models

ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1

Products pickedCatalog numbers

Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN B0BQ1M98FB · quoted

What happenedReviewer label, not graded by code

Adult Crest gel, quantity one, $4.33 with reconciled charges and the saved destination and payment matched. The assistant presented it as a proposed gel and asked before going further, but never established that it was the usual white tube. The first request used unsupported options; the recovery dropped the variant constraint.

Result

Assistant mistake

Reviewer's label partial

$4.33 proposal; usual tube unverified

Next step

Keep a literal product constraint in supported query or variant text after recovering from a schema error; ask about the usual product instead of assuming history.

Test flight

A1 · Amazon

GPT-5.6 Luna

Asked a fair questionappropriate_clarificationAnonymized

Asked which usual tube

Real prompt, scrubbedfrozen before the runs; hash checked for both models

ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept the white-tube query and quantity one. Checkout Bench returned product_choice_requires_user from sparse search titles; Luna showed the real alternatives, pointed out the multipacks and asked which tube is the usual one. No made-up history, cart change or purchase. An honest incomplete result, not proof that no white tube exists.

Result

Asked a fair question

Reviewer's label appropriate_clarification

Asked which usual tube

Next step

None forced. Possible backend improvement: look at product-level evidence before escalating title-only uncertainty.

Test flight

A2 · Amazon

DeepSeek V4.1 Flash

Quoted correctlyquote_with_compatibility_pendingAnonymized

$38.59 mobile kit; iPhone port unconfirmed

Real prompt, scrubbedfrozen before the runs; hash checked for both models

need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one

Follow-up: the phone one, 1 transmitter 1 receiver

Products pickedCatalog numbers

DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for SmartphoneASIN B0H94BWKB8 · quoted

What happenedReviewer label, not graded by code

After the supplied follow-up, quoted the 1 TX + 1 Mobile RX kit at $38.59. The longer checkout title passed the released identity fix; quantity, charges, destination, payment and cart restore all verified. The reply said plainly that it is USB-C, but the shopper never said which iPhone port they have. The first Lightning query was not supported by the request.

Result

Quoted correctly

Reviewer's label quote_with_compatibility_pending

$38.59 mobile kit; iPhone port unconfirmed

Next step

Settle USB-C vs Lightning before treating this as a fully compatible, ready-to-buy pick.

Test flight

A2 · Amazon

GPT-5.6 Luna

Quoted correctlyquote_with_compatibility_pendingAnonymized

$38.59 mobile kit; iPhone port unconfirmed

Real prompt, scrubbedfrozen before the runs; hash checked for both models

need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one

Follow-up: the phone one, 1 transmitter 1 receiver

Products pickedCatalog numbers

DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for SmartphoneASIN B0H94BWKB8 · quoted

What happenedReviewer label, not graded by code

One accepted request picked the same phone kit at $38.59; full checkout title, quantity, charges and cleanup passed. The follow-up was never needed because no question was asked. The reply named USB-C and ruled out Lightning, but the actual iPhone port is still unknown.

Result

Quoted correctly

Reviewer's label quote_with_compatibility_pending

$38.59 mobile kit; iPhone port unconfirmed

Next step

Keep it a conditional quote until the connector is confirmed. An accepted long title is not the same as a verified fit for the buyer's device.

Test flight

A3 · Amazon

DeepSeek V4.1 Flash

Checkout Bench backend bugbackend_false_mismatchAnonymized

Matching 8-count pack refused

Real prompt, scrubbedfrozen before the runs; hash checked for both models

Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept Duracell AA, 8-pack and quantity two. Checkout Bench returned product_choice_requires_user, calling an observed Duracell 8 Count listing a different pack size. The reply explained the conflict and proposed the exact 8-count listing twice (16 total), but asked for another approval it didn't need. No cart add, no purchase.

Result

Checkout Bench backend bug

Reviewer's label backend_false_mismatch

Matching 8-count pack refused

Next step

Trace pack-size validation so per-listing pack count stays separate from order quantity. "8 pack" vs "8 Count" is the observed contradiction; the parser root cause isn't proven yet.

Test flight

A3 · Amazon

GPT-5.6 Luna

Checkout Bench backend bugbackend_false_mismatchAnonymized

Matching 8-count pack refused

Real prompt, scrubbedfrozen before the runs; hash checked for both models

Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Same request and the same wrong 8-pack vs 8 Count refusal as DeepSeek. Luna found two matching 8-count Duracell packs in its reply but asked for another pick instead of reaching a quote. Saying Amazon needed a choice blamed the store for a Checkout Bench validation stop.

Result

Checkout Bench backend bug

Reviewer's label backend_false_mismatch

Matching 8-count pack refused

Next step

Fix the shared pack validation rather than patching two model prompts; quantity two was read correctly.

Test flight

A4 · Amazon

DeepSeek V4.1 Flash

Quoted correctlycorrect_quoteAnonymized

$5.38; hard cap kept

Real prompt, scrubbedfrozen before the runs; hash checked for both models

my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls

Products pickedCatalog numbers

Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 FootASIN B085SBNFNP · quoted

What happenedReviewer label, not graded by code

One 6 ft USB-C to USB-C cable: $4.88 + $0 shipping + $0.50 tax = $5.38, under the $12 hard cap. Exact connector, length and full total; saved destination and payment matched.

Result

Quoted correctly

Reviewer's label correct_quote

$5.38; hard cap kept

Next step

None for this run.

Test flight

A4 · Amazon

GPT-5.6 Luna

Assistant mistakecorrect_product_budget_with_payment_omissionAnonymized

$5.38; preferred card omitted

Real prompt, scrubbedfrozen before the runs; hash checked for both models

my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls

Products pickedCatalog numbers

Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 FootASIN B085SBNFNP · quoted

What happenedReviewer label, not graded by code

Same correct cable and $5.38 total, under the $12 cap. But the shopper context names a preferred card for Amazon; Luna left payment out of its request, so the account's default card came back unmatched. The reply said Amazon didn't match the preferred card, though no such selection was ever requested or tested.

Result

Assistant mistake

Reviewer's label correct_product_budget_with_payment_omission

$5.38; preferred card omitted

Next step

Carry known preferences into the request, and tell an account default apart from a tested payment mismatch. An assistant omission, not evidence the store rejected the card.

Test flight

A5 · Amazon

DeepSeek V4.1 Flash

Assistant mistakeunsupported_compatibility_claimAnonymized

$16.42; unsupported compatibility claim

Real prompt, scrubbedfrozen before the runs; hash checked for both models

add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together

Products pickedCatalog numbers

Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN B0BQ1M98FB · quoted
Oral-B Deep Clean Replacement Brush Heads, 2 CountASIN B003UBTYKI · quoted

What happenedReviewer label, not graded by code

Both items in one reconciled $16.42 basket, cleanup done. The heads were Oral-B Deep Clean battery-brush replacements; the buyer's handle type was unknown. The reply claimed they are standard refills for rechargeable handles with no data to back it, and the request dropped the "regular electric brush" detail.

Result

Assistant mistake

Reviewer's label unsupported_compatibility_claim

$16.42; unsupported compatibility claim

Next step

Keep compatibility constraints through search and find out the handle type, or ask. A ready checkout alone doesn't approve the pick.

Test flight

A5 · Amazon

GPT-5.6 Luna

Assistant mistakecompatibility_unconfirmedAnonymized

$20.61; handle fit unconfirmed

Real prompt, scrubbedfrozen before the runs; hash checked for both models

add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together

Products pickedCatalog numbers

Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN B0BQ1M98FB · quoted
Oral-B Daily Clean Replacement Brush Heads, Pack of 3ASIN B089LBBDZR · quoted

What happenedReviewer label, not graded by code

Kept the regular electric-brush detail and chose Daily Clean 3-pack heads with the Crest in one $20.61 basket, cleanup done. Still didn't establish the handle type or caveat fit. The reply labeled a 10:15 PM local expiry as UTC (it was 05:15 UTC). One refused out-of-scope read is in the record; the quote itself worked.

Result

Assistant mistake

Reviewer's label compatibility_unconfirmed

$20.61; handle fit unconfirmed

Next step

Confirm the handle type or qualify the pick; fix the time-zone wording separately.

Test flight

A6 · Amazon

DeepSeek V4.1 Flash

Assistant mistakesoft_budget_misclassifiedAnonymized

$88.85; soft target made hard

Real prompt, scrubbedfrozen before the runs; hash checked for both models

ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50

Products pickedCatalog numbers

STANLEY Quencher H2.0 FlowState Tumbler (Peony, 40 oz)ASIN B0CZF97C8N · quoted

What happenedReviewer label, not graded by code

Pink Peony Stanley 40 oz, full quote $88.85, above target; arrival Oct 27, before Saturday Oct 31; seller, charges, destination and payment shown; cart restore verified. The assistant sent the budget as a hard $50 cap, unlike the tester's soft-target note, stopped after one search and asked about other pinks though any pink was allowed.

Result

Assistant mistake

Reviewer's label soft_budget_misclassified

$88.85; soft target made hard

Next step

Keep target vs hard cap apart from store over-budget behavior; search within budget before asking for a preference already given. The wording is somewhat ambiguous, so one run doesn't prove a general budget bug.

Test flight

A6 · Amazon

GPT-5.6 Luna

Checkout Bench backend bugidentity_block_and_reply_gapsAnonymized

$29.97 item; longer-title block

Real prompt, scrubbedfrozen before the runs; hash checked for both models

ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50

Products pickedCatalog numbers

STANLEY Quencher H2.0 Flow State Tumbler, 40 oz, PeonyASIN B0CRMP3RQT · selected; Checkout Bench blocked on title

What happenedReviewer label, not graded by code

Saw a pink Peony 40 oz at $29.97 item price, but Checkout Bench blocked it because the checkout title was longer than the selected title at the same price. The released DJI fix passes; this different title case is still open. No full charges or arrival shown. The reply called it under $50 from the item price, left out Saturday, and treated the soft target as hard. Cart restore verified.

Also: Reply also called it under budget from item price alone and dropped the Saturday deadline.

Result

Checkout Bench backend bug

Reviewer's label identity_block_and_reply_gaps

$29.97 item; longer-title block

Next step

Read the retained product-identity evidence for this title block before changing the identity rule again; keep the deadline and separate item price from delivered total.

Test flight

U1 · Uber Eats

DeepSeek V4.1 Flash

Asked a fair questionappropriate_address_clarificationAnonymized

Asked which saved address

Real prompt, scrubbedfrozen before the runs; hash checked for both models

yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept the Jumbo Jack meal, medium Coke and tip. The saved address label in the request didn't match the label on the account, and the assistant asked which saved address to use, with no made-up answer. No menu, price, tip or quote proved; no cart change.

Result

Asked a fair question

Reviewer's label appropriate_address_clarification

Asked which saved address

Next step

A durable buyer alias or evidence-based address match could avoid this question; don't guess a destination from unrelated store context.

Test flight

U1 · Uber Eats

GPT-5.6 Luna

Assistant mistakeunverified_home_and_store_unavailableAnonymized

Searched an unverified default address

Real prompt, scrubbedfrozen before the runs; hash checked for both models

yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept the meal, medium Coke and tip but left out the destination. Checkout Bench searched the account default and returned store_not_found. The reply said that was the requested saved address without evidence the labels matched; nearby results don't prove availability at the intended address. No quote, tip or cart change.

Result

Assistant mistake

Reviewer's label unverified_home_and_store_unavailable

Searched an unverified default address

Next step

Resolve the intended saved address before blaming store availability on it; don't switch restaurant or address without asking.

Test flight

U2 · Uber Eats

DeepSeek V4.1 Flash

Stopped by operatorunsupported_configuration_and_retry_loop_operator_stoppedAnonymized

Operator stopped after 5 attempts

Real prompt, scrubbedfrozen before the runs; hash checked for both models

can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

The first request correctly listed the two large fries as separate configurations, one no-salt. Checkout Bench refuses identical items with different options (duplicate_product). The assistant then tried four single no-salt requests; all returned item_dialog_missing, so no-salt availability stayed unknown. The operator stopped the loop after five accepted quotes; no final reply. Later reads confirmed all five blocked and settled, no cart adds, VM stopped.

Also: Checkout Bench can't hold two of the same item with different options (duplicate_product).

Result

Stopped by operator

Reviewer's label unsupported_configuration_and_retry_loop_operator_stopped

Operator stopped after 5 attempts

Next step

Separate configured line identity from product identity, and stop unchanged retries after a repeated reader failure. Not an unassisted finish.

Test flight

U2 · Uber Eats

GPT-5.6 Luna

Checkout Bench backend bughonest_unsupported_with_attribution_gapAnonymized

Different options unsupported

Real prompt, scrubbedfrozen before the runs; hash checked for both models

can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept two distinct salt configurations, hit the same duplicate_product limit and stopped after one request without merging or dropping the restriction. The reply blamed Uber Eats rather than Checkout Bench and offered two no-salt fries without evidence that option exists. No cart adds or quote.

Also: Reply blamed Uber Eats instead of Checkout Bench and offered an unverified alternative.

Result

Checkout Bench backend bug

Reviewer's label honest_unsupported_with_attribution_gap

Different options unsupported

Next step

Explain Checkout Bench's limit accurately and label unverified alternatives as such; support configuration-aware line identity in the backend.

Test flight

U3 · Uber Eats

DeepSeek V4.1 Flash

Stopped by operatorrepeated_menu_store_failures_operator_stoppedAnonymized

Operator stopped after 7 attempts

Real prompt, scrubbedfrozen before the runs; hash checked for both models

pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

The first request kept both large pizzas, but menu matching failed. Seven accepted quotes alternated product_choice_requires_user and store_not_found as the query and restaurant name changed. Recovery dropped the sizes for a while and once shrank to "pizza, quantity two", losing the toppings. The operator stopped the loop; no final reply, basket or deal. Later reads confirmed all seven blocked and settled, no cart steps, VM stopped.

Also: Menu matching failed repeatedly on Checkout Bench's side.

Result

Stopped by operator

Reviewer's label repeated_menu_store_failures_operator_stopped

Operator stopped after 7 attempts

Next step

Show the actual observed menu and availability evidence and keep the original constraints while probing; stop repeated failures without inventing a pizza or deal.

Test flight

U3 · Uber Eats

GPT-5.6 Luna

Assistant mistakemenu_match_block_with_unverified_alternativeAnonymized

Menu block; unobserved alternative suggested

Real prompt, scrubbedfrozen before the runs; hash checked for both models

pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept both large pizzas, then stopped after one product_choice_requires_user refusal for the pepperoni. No cheese search, deal or checkout. The reply admitted the quote was incomplete but proposed a specifically named Large 1-Topping Pizza that wasn't in what Checkout Bench returned. No cart activity.

Also: Checkout Bench also stopped on a menu match for the pepperoni.

Result

Assistant mistake

Reviewer's label menu_match_block_with_unverified_alternative

Menu block; unobserved alternative suggested

Next step

Return observed menu candidates and keep going on independent items instead of inventing a plausible menu option.

Test flight

I1 · Instacart

DeepSeek V4.1 Flash

Quoted correctlycorrect_quote_cheapest_delivered_unprovenAnonymized

$26.92; store discovery worked

Real prompt, scrubbedfrozen before the runs; hash checked for both models

Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you

Products pickedCatalog numbers

Coca-Cola Soda, 24 x 12 fl ozInstacart product 93587 · quoted

What happenedReviewer label, not graded by code

Found local offers, chose Walmart's 24 x 12 fl oz regular Coke cans ($15.52 merchandise) and kept the product and store link through the follow-up. Reconciled checkout $26.92 including service, long-distance, tax and fee lines plus tip. Walmart had the lowest matching product price, but other stores' full fees weren't quoted, so cheapest delivered total is unproven.

Result

Quoted correctly

Reviewer's label correct_quote_cheapest_delivered_unproven

$26.92; store discovery worked

Next step

Keep product-price comparison separate from delivered-total comparison.

Test flight

I1 · Instacart

GPT-5.6 Luna

Assistant mistakeunnecessary_retailer_clarificationAnonymized

Asked for a store without searching

Real prompt, scrubbedfrozen before the runs; hash checked for both models

Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Sent no search or quote. Asked the shopper to name a retailer and said Instacart needs one, though the single-item store-discovery path was available (DeepSeek used it in the previous run). Misses the prewritten outcome of finding the cheapest store. No cart activity. DeepSeek read the Instacart-specific note; Luna read only the general files.

Result

Assistant mistake

Reviewer's label unnecessary_retailer_clarification

Asked for a store without searching

Next step

Line up guide use with what the tool can actually do; ask only for missing preferences. Check what the assistant read before blaming only the model.

Test flight

I2 · Instacart

DeepSeek V4.1 Flash

Checkout Bench backend bugconflicting_pack_evidence_unresolvedAnonymized

Same-store items; 12 vs 4 egg-count conflict

Real prompt, scrubbedfrozen before the runs; hash checked for both models

running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees

Products pickedCatalog numbers

Marketside Grade A Cage Free Large Brown Eggs, 12 ctInstacart product 20659148 · selected; blocked before cart
Great Value 2% Reduced Fat Milk, 128 ozInstacart product 20630136 · selected; blocked before cart

What happenedReviewer label, not graded by code

Found both items at one Walmart and kept exact product links for the follow-up. The egg offer says 12 ct, the milk 128 oz. The follow-up called the same egg listing a pack of four and blocked before adding to cart, so the real pack size is unresolved. The assistant repeated "four" as fact and proposed quantity three. Milk found in search only, not verified at checkout; no basket or fees.

Also: The assistant repeated the disputed pack count as fact.

Result

Checkout Bench backend bug

Reviewer's label conflicting_pack_evidence_unresolved

Same-store items; 12 vs 4 egg-count conflict

Next step

Trace pack-count reading against the retained source evidence before suggesting three packs; tell a candidate basket apart from a verified checkout.

Test flight

I2 · Instacart

GPT-5.6 Luna

Assistant mistakepremature_retailer_clarificationAnonymized

Asked for a store without searching

Real prompt, scrubbedfrozen before the runs; hash checked for both models

running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Asked which store to use for both items without any search or quote attempt, though same-store discovery was available (DeepSeek used it in the previous run). Nothing dropped or falsely quoted, but an extra interruption before trying the tool.

Result

Assistant mistake

Reviewer's label premature_retailer_clarification

Asked for a store without searching

Next step

Use store discovery before turning a tool-solvable store choice into a required question; an optional preference can be offered without blocking.

Test flight

W1 · Walmart

DeepSeek V4.1 Flash

Quoted correctlycorrect_pickup_quote_pending_bag_feeAnonymized

$5.17 pickup; bag fee pending

Real prompt, scrubbedfrozen before the runs; hash checked for both models

can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway

Products pickedCatalog numbers

Great Value Purified Drinking Water, 16.9 fl oz, 24 CountWalmart item 13448854 · quoted

What happenedReviewer label, not graded by code

One Great Value 24-count water pack, pickup as requested. Reconciled quote $5.17 ($3.97 items + $1.20 taxes and fees). The reply disclosed the pending bag fee and listed a $5.46 authorization-hold amount separately. A pickup quote, not a booking or a placed hold.

Result

Quoted correctly

Reviewer's label correct_pickup_quote_pending_bag_fee

$5.17 pickup; bag fee pending

Next step

Keep pending fees and a possible future hold separate from the quote and any real charge.

Test flight

W1 · Walmart

GPT-5.6 Luna

Quoted correctlycorrect_pickup_quote_pending_bag_feeAnonymized

$5.17 pickup; bag fee pending

Real prompt, scrubbedfrozen before the runs; hash checked for both models

can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway

Products pickedCatalog numbers

Great Value Purified Drinking Water, 16.9 fl oz, 24 CountWalmart item 13448854 · quoted

What happenedReviewer label, not graded by code

The same Great Value 24-count water pack for pickup, $5.17 reconciled estimate. The reply disclosed the pending bag fee and called it a quote; no shipping swap or booking claim.

Result

Quoted correctly

Reviewer's label correct_pickup_quote_pending_bag_fee

$5.17 pickup; bag fee pending

Next step

No correction needed; keep the pending-fee caveat before buying.

Test flight

W2 · Walmart

DeepSeek V4.1 Flash

Quoted correctlybrand_pickup_correct_pack_ambiguousAnonymized

$3.26 pickup; 2 double rolls

Real prompt, scrubbedfrozen before the runs; hash checked for both models

get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup

Products pickedCatalog numbers

Great Value Everyday Strong Paper Towels, 2 Double RollsWalmart item 10762707702 · quoted

What happenedReviewer label, not graded by code

Switched Bounty to Great Value and kept pickup. Chose one 2 Double Rolls package at $3.26 with tax, with the pending bag fee disclosed. The request dropped the original "big pack" wording, so the intended size is unclear; no made-up size.

Result

Quoted correctly

Reviewer's label brand_pickup_correct_pack_ambiguous

$3.26 pickup; 2 double rolls

Next step

Keep a soft pack-size wish while changing brands and show the pack chosen; don't invent a number from "big pack".

Test flight

W2 · Walmart

GPT-5.6 Luna

Asked a fair questionobserved_pack_clarificationAnonymized

Asked among observed pack choices

Real prompt, scrubbedfrozen before the runs; hash checked for both models

get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Kept the switch to Great Value, pickup and the "big pack" wish. Checkout Bench found 12 triple, 6 roll and 2 roll choices but needed an exact pack; Luna asked among those real options, with no made-up size or shipping fallback. No cart activity. Meets the prewritten allowance for a pack question.

Result

Asked a fair question

Reviewer's label observed_pack_clarification

Asked among observed pack choices

Next step

Consider picking from observed packs for soft size wishes instead of treating a vague size as exact.

Test flight

T1 · Target

DeepSeek V4.1 Flash

Assistant mistakehonest_search_block_unverified_suggestionsAnonymized

42-count unverified; shipping minimum unknown

Real prompt, scrubbedfrozen before the runs; hash checked for both models

target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Two requests kept Tide PODS 42-count and shipping; both returned product_choice_requires_user with no candidates. The assistant correctly said search failed before cart, so minimum, fees and total are unknown, with no filler or pack change. But it added unverified claims about Target's usual pack counts and unneeded internal preference details. No cart activity.

Also: Checkout Bench also returned no candidate listings.

Result

Assistant mistake

Reviewer's label honest_search_block_unverified_suggestions

42-count unverified; shipping minimum unknown

Next step

Return observed listings or source-backed reasons for no match; avoid catalog guesses.

Test flight

T1 · Target

GPT-5.6 Luna

Asked a fair questionhonest_product_match_blockAnonymized

42-count unverified; shipping minimum unknown

Real prompt, scrubbedfrozen before the runs; hash checked for both models

target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

One request kept Tide PODS 42-count and shipping. Checkout Bench found no fitting listing and returned no alternatives. The assistant said no price or minimum was verified, didn't claim it was out of stock, and asked before changing pack or quantity. No cart activity.

Result

Asked a fair question

Reviewer's label honest_product_match_block

42-count unverified; shipping minimum unknown

Next step

Show the candidates Checkout Bench inspected and why they didn't match.

Test flight

C1 · Chewy

DeepSeek V4.1 Flash

Assistant mistakeexact_bag_unresolved_one_time_unexercisedAnonymized

30 lb bag unresolved; buy-once untested

Real prompt, scrubbedfrozen before the runs; hash checked for both models

dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

Two attempts kept Chicken & Rice and 30 lb; the second added Complete Essentials as a search refinement. Both blocked on product match, showing senior 7+ or other flavors. The existing cart was set aside and restored both times; no cart add or Autoship. The reply asked before substituting but wrongly said the shopper named Complete Essentials, and described the buy-once policy without reaching that checkout step.

Also: Checkout Bench blocked both attempts on product match.

Result

Assistant mistake

Reviewer's label exact_bag_unresolved_one_time_unexercised

30 lb bag unresolved; buy-once untested

Next step

Resolve the exact formula and weight from observed choices or a product link; don't credit the shopper with an inferred product line.

Test flight

C1 · Chewy

GPT-5.6 Luna

Asked a fair questionexact_bag_unresolved_one_time_unexercisedAnonymized

30 lb bag unresolved; buy-once untested

Real prompt, scrubbedfrozen before the runs; hash checked for both models

dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that

Products pickedCatalog numbers

No product picked; not recorded.

What happenedReviewer label, not graded by code

One request kept the exact adult Chicken & Rice 30 lb target. Checkout Bench showed senior 7+ or other flavors but no exact bag. The assistant left it unresolved and asked before changing anything, without claiming a quote or buy-once check. Existing cart set aside and restored; no cart add or Autoship.

Result

Asked a fair question

Reviewer's label exact_bag_unresolved_one_time_unexercised

30 lb bag unresolved; buy-once untested

Next step

Coverage gap: a verified exact product is still needed to test one-time checkout.

Leaderboard

Pass rate per model per store. A run passes only if every check passes: right item, right quantity, under the price cap, and asked a question when the request was unclear.

Task set: messy-240 Round: Oct 2026

All stores · 240 requests per model

#	Model	Overall	Amazon	Walmart	Target	Instacart	Uber Eats	Chewy	Runs	Median steps
1	Model A	62%	71%	64%	58%	49%	66%	62%	240	31
2	Model B	55%	63%	59%	61%	40%	52%	57%	240	38
3	Model C	47%	55%	48%	44%	37%	50%	46%	240	42
4	Model D	33%	41%	39%	33%	22%	35%	30%	240	57
5	Model E	25%	29%	31%	24%	18%	27%	21%	240	66
6	Model F	11%	12%	15%	9%	6%	14%	10%	240	81

Most common reasons for failing

Check	Share of failures	Typical case
Right item	38%	Picked a different kit, size or flavor than asked
Asked when ambiguous	27%	Several products matched and the agent guessed
Quantity	14%	“2 of those” became 1, or a 2-pack became 2 two-packs
Under price cap	12%	Ignored fees when the request said “under 20 bucks w/ fees”

Task bank

Requests written the way people really type, each with the answer written down before any model sees it: which products count as correct, the quantity, the price cap, and whether the right move is to ask a question first.

Store: all Tag: all Add task

12 of 240 tasks shown

ID	Store	What the shopper typed	Counts as correct	Tags	Difficulty
T-001	Amazon	“can you get me the dji mic mini thing from amazon”	Any DJI Mic Mini kit after asking which kit	ambiguousvariant	hard
T-014	Walmart	“need paper towels the big pack, bounty not the off brand walmart pls”	Bounty Select-A-Size, 12 double rolls or larger	brandsize	medium
T-022	Chewy	“dog food same as last time?? the purina pro plan salmon one, 30lb”	Purina Pro Plan Adult Salmon & Rice, 30 lb	exactsize	easy
T-031	Target	“get 2 of those owala water bottles 24oz, the blue-ish one my kid likes”	Owala FreeSip 24 oz, any blue colorway, qty 2	quantitycolor	medium
T-040	Instacart	“bananas (not too green lol), 2 gal whole milk, eggs 18ct, whatever bread is cheapest”	4 items; cheapest sandwich loaf at that store	multi-itemcheapest	hard
T-047	Uber Eats	“chipotle bowl chicken double rice no beans, under 20 bucks w/ fees”	Chicken bowl, extra rice, no beans; total ≤ $20	modifiersbudget	hard
T-055	Amazon	“my airpods case died can u find just the charging case, gen 2 usb c”	Apple charging case only, USB-C; ask if no case-only listing exists	ambiguousaccessory	hard
T-063	Target	“tide pods free & clear like 80ish count”	Tide PODS Free & Clear, 72–91 count	rangevariant	medium
T-070	Amazon	“ink for my HP envy 6055e, black AND color”	HP 67 or 67XL black + tri-color	compatibilitymulti-item	medium
T-078	Chewy	“cat litter that doesnt smell, unscented clumping, 40 lb-ish, under $30”	Unscented clumping, 35–42 lb, ≤ $30	budgetrange	medium
T-086	Target	“huggies sensitive wipes, 3 packs, whatever count is the better deal”	Huggies Sensitive, lowest price per wipe, 3 packs	unit pricequantity	hard
T-093	Amazon	“uh the ninja air fryer thats like 100 bucks, 4qt?”	Ninja 4 qt air fryer, $80–120; ask if several match	vaguebudget	medium