---
title: "Test suite · Checkout Bench"
description: "Checkout Bench test suite: recent real Checkout Bench runs with condensed timelines (scrubbed), plus a leaderboard and task bank."
canonical: https://www.checkoutbench.ai/research
last_updated: 2026-10-25
---

## Methodology

The Checkout Bench benchmark measures whether an AI shopping agent can take a person's request and reach a ready checkout at a real store. Each run happens in a fresh cloud browser inside real, signed-in store accounts, using the kinds of requests people actually type.

Runs are graded in code from the run's record, and no model grades another model. A run passes only when the agent picked the right item in the right quantity, stayed within any budget the person gave, and asked a question when the request was really unclear. An order is placed only after the person says yes to the checked total, and a purchase counts once the store confirms the order.

The stores covered are Amazon, Uber Eats, Instacart, Walmart, Target and Chewy. The assistant models in these results are GPT-5.6 Sol, DeepSeek V4.1 Flash, GPT-5.6 Luna, Claude Opus 4.8, Grok 4.7 and Kimi K3.

This page was last updated on November 6, 2026.

## Key findings

- **35%** of 527 real checkout requests ended in a ready checkout. The biggest drop is from search results to checkout read: 518 to 231. (Figure 1)
- **11 of 30** reviewed runs went wrong on the assistant's side, 5 on Checkout Bench's, and 7 quoted correctly. (Figure 2)
- **40** stores and sites can be connected today, and 7 return a full cart and total. (Section 04)

## Can an AI agent buy from your store? (for retailers)

**We send shopping agents to your site and show you where they get stuck.**

People are starting to hand their shopping to AI assistants. When an assistant can't find the right size, gets lost in a pop-up or stalls at checkout, the sale just doesn't happen, and nobody tells you. We find those places before your customers do.

### Where agents get stuck

A flow of who runs and where they go shows four lab models handing a task to Checkout Bench's agent, which fans out to the six stores and returns each store's real ready-checkout count, median checkout total and median time.

- **Search.** The agent searches the way a person talks and gets nothing useful back. What we check: Do ordinary requests land on the right product?
- **Product options.** Size, color or pack count is chosen in a way agents can't read, so they stop or pick the wrong one. What we check: Can an agent pick the variant the person asked for, even when the store calls “green” “Lime”?
- **Sign-in.** Logins, pop-ups and account checks block the agent before it reaches the cart. What we check: Does a signed-in session get through without a person stepping in?
- **Cart.** Quantities, bundles and add-ons end up different from what was asked. What we check: Does the cart hold exactly the right item and amount?
- **Checkout.** Fees, delivery choices or a final review screen the agent can't get past. What we check: Does it reach the final review screen with the right total?

### What you get back

- A score for each step, from search to checkout
- A replay of every place an agent got stuck (coming soon)
- A fix list, ordered by how many runs each problem blocked
- Every result split into “your site” and “the agent”, so you only fix what's yours
- Re-checks: After you ship fixes we run the same requests again, and on a schedule after that. If a site change starts breaking agents, you hear about it with the step that broke.

### A private test environment built on your store

1. We work with your team and your own product data: your catalog, your variants and your real checkout flow.
2. We build a test setup that is yours alone, with the requests your customers actually make.
3. We run the assistants your customers are starting to use, such as ChatGPT, Gemini and Perplexity, through it and show exactly where each one gets stuck.
4. Nothing is ever bought, and your data stays in your environment.

**Why trust the numbers.** The same runs power our public benchmark. Every run happens in a fresh cloud browser on a real store and is graded in code, not by another AI.

**For developers.** The engine that drives these runs is also available as an API for teams building assistants that shop. Details are further down the page.

## What exists today (real)

- **40**: stores and sites a person can connect an account to today: 12 built in, 28 set up from their web address
- **7**: stores where a full cart and total come back today: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE
- **1**: fresh cloud browser per run, thrown away afterwards
- **384**: trace events in one real run: DeepSeek V4.1 Flash asked Amazon for a DJI mic; Checkout Bench blocked all five quotes as product mismatch

## Charts on this page

- **Quote funnel (real):** Checkout requested 527 (100%) → Browser ready 524 (99%) → Signed in 518 (98%) → Search results 518 (98%) → Checkout read 231 (44%) → Quote checked 196 (37%) → Ready Checkout 186 (35%). Source: [Checkout Bench Research](https://www.checkoutbench.ai/research).
- **Whose problem it was, by store (real, Test flight, 30 runs):** Amazon (12): Quoted correctly 3, Asked a fair question 1, Assistant mistake 5, Checkout Bench backend bug 3, Stopped by operator 0; Walmart (4): Quoted correctly 3, Asked a fair question 1, Assistant mistake 0, Checkout Bench backend bug 0, Stopped by operator 0; Target (2): Quoted correctly 0, Asked a fair question 1, Assistant mistake 1, Checkout Bench backend bug 0, Stopped by operator 0; Instacart (4): Quoted correctly 1, Asked a fair question 0, Assistant mistake 2, Checkout Bench backend bug 1, Stopped by operator 0; Uber Eats (6): Quoted correctly 0, Asked a fair question 1, Assistant mistake 2, Checkout Bench backend bug 1, Stopped by operator 2; Chewy (2): Quoted correctly 0, Asked a fair question 1, Assistant mistake 1, Checkout Bench backend bug 0, Stopped by operator 0.
- **Lab runs per day (real):** Sep 26 76, Sep 27 39, Sep 28 12, Sep 29 25, Oct 1 264, Oct 2 13, Oct 25 29; no runs on the other days.
- **Model rank by store:** GPT-5.6 Luna 2/1/2/3/1/2; DeepSeek V4.1 Flash 1/2/3/1/2/1; Grok 4.7 3/4/1/2/4/3; Gemini 3.8 Flash 4/3/4/4/3/4 (ranks at Amazon, Walmart, Target, Instacart, Uber Eats, Chewy).
- **Price check ladder (the real KW-color-01 case):** reference $349.00, grader allowance ±$6.98 = max(2% of the pre-tax item subtotal, 50¢).
- **Quoted totals through a day of checks (KW-cable-01 case):** totals near $28.12 until a $2.99 shipping fee at 15:00 pushes every quote over the $30 cap.

## Five directions

### Option A (Benchmark): How well do AI agents actually shop?

We hand models the requests real people type, and let them work inside real, signed-in store accounts at Amazon, Walmart, Target and four more. Code checks the final cart. We publish what passed and what didn't.

- **Who buys it:** Model labs and agent companies that want an outside number for shopping tasks, and the people choosing between models.
- **What they get:** A public leaderboard by model and by store; Private runs on models that aren't released yet; The full step-by-step trace behind every pass and fail
- **How it works:** A request written the way people write: “can you get me the dji mic mini thing from amazon” → The agent works in a fresh cloud browser on the real store, all the way through checkout → Code checks the item, the quantity and the price cap
- **Pricing idea:** Leaderboard is free to read. Private evaluation rounds are priced per model, per round (not set yet).

### Option B (RL training environment): A practice store for training shopping agents

Recorded copies of real store pages that reset instantly, a bank of messy requests with the right answers written down, and graders written in code. For labs training agents that buy things.

- **Who buys it:** Research and post-training teams at model labs.
- **What they get:** A container with a reset / step interface that runs many copies in parallel; Labeled tasks: each one lists the products that count as correct, or says the right move is to ask; Graders in code, no model judging another model; A held-out set of live runs to check the copy still matches the real store
- **How it works:** Record real sessions at each store, with personal details removed → Replay them as a store that resets in milliseconds and behaves like the original → Score every episode in code and keep the live exam separate from training
- **Pricing idea:** Licensed as task packs. Comparable deals we have seen land around $200–2,000 per task depending on difficulty and exclusivity. A range, not a quote.

### Option C (For retailers): Can an AI agent buy from your store?

We send shopping agents to your site with ordinary requests and show you where they get stuck: search, product options, sign-in, cart or checkout.

- **Who buys it:** E-commerce and growth teams at retailers and brands, and the platforms that host their stores.
- **What they get:** A score for each step from search to checkout; A recording of every place an agent failed; A fix list ordered by how many runs each problem blocked
- **How it works:** Pick 20–50 requests your customers really make → Several different agents try each one in fresh browsers → Read the report, fix things, run it again
- **Pricing idea:** A flat fee per audit, with an optional monthly re-check.

### Option D (Monitoring): Know when a store change breaks shopping agents

We run a fixed set of shopping tasks against each store on a schedule. When a site change makes them fail, you get an alert with the step that broke and a recording of it.

- **Who buys it:** Teams that ship shopping agents or browser automation, and retailers who want agents to keep working on their site.
- **What they get:** Scheduled runs per store; Alerts by email, Slack or webhook; A before-and-after comparison of the step that broke
- **How it works:** Choose the stores and tasks you care about → We run them every few hours in fresh browsers → When something that passed starts failing, you get the step, the screenshot and the difference
- **Pricing idea:** Monthly, per store watched.

### Option E (API): One API to sign in, fill a cart and check out

Your agent sends what the person asked for and which connected store to use. We sign in, find the product, build the cart and send back the total. The order is placed when the person says yes.

- **Who buys it:** Developers building assistants that need to buy things for their users.
- **What they get:** Account connection at 40 stores and sites; Cart and total at 7: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE; Orders placed only after the person confirms; A full trace of every request
- **How it works:** The person connects a store account once → Your agent calls the API with their request → You get the cart and total back, and place the order only after they confirm
- **Pricing idea:** Paid per cart total returned and per order placed (rates not set yet).

## FAQ

### Is this a real product?

Yes. Checkout Bench runs AI shopping agents through real, signed-in store checkouts and shows where they get stuck. Retailers can request a private test of their store, the test suite at /lab lets signed-in users run assistants against real quotes, and agent developers can use Checkout Bench's API, documented under docs.

### What exists today?

Account connection at 40 stores and sites (12 built in, 28 set up from their web address); a cart and total at 7 (Amazon, Uber Eats, Instacart, Walmart, Target, Chewy and DETOURE), and at the first six the order itself once the person says yes; a fresh cloud browser for every run; a full trace of each run; and the test suite at /lab. One real run, DeepSeek V4.1 Flash asked for a DJI mic on Amazon, recorded 384 trace events, and all five quotes were blocked as product mismatch (the checkout title was longer than the product page's).

### How is a run graded?

In code, from the run's record: right item, right quantity, within any budget the person gave, and a question when the request was really unclear. A purchase counts only once the store confirms the order. No model grades another model.

### When does an order get placed?

When a person says yes. A purchase needs the person's explicit yes to that quote's total, and it counts only once the store confirms the order.

### Are the numbers on these pages real?

The facts, charts and recorded runs are real and marked Real.

# Test suite

> Recent runs are real (scrubbed).

## Leaderboard

| Model | Amazon | Walmart | Target | Instacart | Uber Eats | Chewy |
|---|---|---|---|---|---|---|
| Model A | 71% | 64% | 58% | 49% | 66% | 62% |
| Model B | 63% | 59% | 61% | 40% | 52% | 57% |
| Model C | 55% | 48% | 44% | 37% | 50% | 46% |
| Model D | 41% | 39% | 33% | 22% | 35% | 30% |
| Model E | 29% | 31% | 24% | 18% | 27% | 21% |
| Model F | 12% | 15% | 9% | 6% | 14% | 10% |

## Charts (real)

Source: [Checkout Bench Research](https://www.checkoutbench.ai/research).

- Lab runs per day (PDT): Sep 26 76, Sep 27 39, Sep 28 12, Sep 29 25, Oct 1 264, Oct 2 13, Oct 25 29
- Run outcomes (596 assistant runs): Quote ready 168, Quote blocked 240, Asked the person 95, No quote 62, Other 31
- Quotes by store (ready/blocked/preparing): Amazon 101/87/0, Uber Eats 34/95/2, Instacart 23/78/1, Walmart 13/41/2, Chewy 10/17/0, Target 5/18/1
- Top blockers: product_choice_requires_user 140, product_mismatch 19, checkout_link_missing 12, item_dialog_missing 12, below_store_minimum 12, store_not_found 12, item_choices_required 11, quote_failed 10, over_budget 10, checkout_not_ready 9
- Seconds to a quote (median, p90): Walmart 34.3/74.5, Amazon 34.6/60.7, Instacart 39.3/49.6, Uber Eats 46.2/117.8, Target 56.8/115.7, Chewy 63.1/90.3
- Median seconds to stage: Browser ready 8.2, Signed in 16.2, Search results 23.3, Product chosen 26.4, Checkout read 39.7, Quote checked 43.9
- Assistant models (runs, median events, median seconds, median model cost USD): GPT-5.6 Sol 381, 77, 47.2, 0.0639; DeepSeek V4.1 Flash 129, 95, 64.0, 0.0068; GPT-5.6 Luna 51, 105, 78.9, 0.0047; Claude Opus 4.8 10, 85.5, 64.4, 0.0924; Grok 4.7 10, 82, 56.0, 0.0116; Kimi K3 9, 84, 58.4, 0.0295
- Ready Checkout totals (median, min to max USD): Amazon 11.0 (4.33 to 220.5), Uber Eats 26.8 (15.22 to 44.91), Instacart 26.8 (16.97 to 46.29), Walmart 13.1 (4.88 to 21.31), Chewy 9.3 (8.25 to 40.06), Target 40.5 (8.58 to 40.53)
- Listing vs quoted: DJI Mic Mini (2 TX + 1 RX) 79.00 to 87.10, Crest Pro-Health toothpaste 4.3 oz 3.93 to 4.33
- Tests passing: Cloud 436/436, Lab 170/170, Service 140/140, Onboarding 18/18
- Activity entries per day: Sep 17 29, Sep 18 18, Sep 22 14, Sep 23 38, Sep 25 3, Sep 26 28, Sep 27 42, Sep 28 28, Sep 29 13, Oct 2 24, Oct 25 8
- Browser replays: placeholders only (16:9, 9:16, 4:3); runs keep event logs, not page recordings.


## Recent runs (real, scrubbed)

### Amazon · Reached order review (Real · Oct 26, 2026, 4:12 PM PDT (ran 41.2 s))

- Request: "USB-C to USB-C charging cable, 6 ft, quantity 1, under $15"
- Assistant: typesafe-ai/jev; provider: Not recorded; Checkout Bench v3 Amazon quote driver (signed-in cloud Chrome); price: $5.38
- Signed-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. Score 1.0.
- Timeline:
  - 0 s: [start] Signed-in cloud Chrome session
  - [search] Right item found: Amazon Basics USB-C to USB-C cable, 6 ft
  - [check] Reached order review: $5.38 total vs $15 cap
  - 41.2 s: [done] Order review read with the right item and total
- What the record shows:
  - Items: Amazon Basics USB-C to USB-C cable, 6 ft × 1
  - Right item: Pass
  - Reached review: Pass
  - Under price cap: $5.38 vs $15
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Best Buy · Found product, no cart or checkout (Real · Oct 26, 2026, 4:45 PM PDT (ran 35.9 s))

- Request: "USB-C to USB-C charging cable, 6 ft, quantity 1, under $15"
- Assistant: No assistant (direct tool calls); provider: n/a; Pomerado Best Buy tool (search + product details), called directly; price: $14.95 before tax
- Signed-out run that found a 6 ft USB-C to USB-C cable listed at $14.95 before tax, under the $15 cap, with free shipping by Oct 30. The Pomerado tool has no cart step, so no cart or order total was seen. The checks below cover only what a product page can show.
- Timeline:
  - 0 s: [start] Pomerado Best Buy tool, signed out, shipping ZIP set
  - 20.0 s: [search] 20 listings; right item found: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, $14.95
  - 35.9 s: [read] Product details: SKU 10555185, $14.95, Shipping FREE Fri, Oct 30; pickup not listed
  - 35.9 s: [done] Product quote read; this tool has no cart step
- What the record shows:
  - Items: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, Black (SKU 10555185) × 1
  - Right item: Pass
  - Under price cap: $14.95 vs $15 (item price, before tax)
  - In stock or pickup: Free shipping, arrives Fri, Oct 30; pickup not listed
  - Reached review: No: the tool has no cart or checkout step
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Amazon · Ready Checkout (Real · Oct 25, 2026, 9:12 PM PDT)

- Request: "Re-run of the last DJI request from the run below, on the exact kit the person had chosen"
- Assistant: No assistant (direct engine check); provider: n/a; Candidate engine, before release; price: $87.10
- The same DJI Mic Mini (2 TX + 1 RX) that the live engine had blocked as product_mismatch, re-quoted with the title-alias fix.
- Timeline:
  - 0 s: [start] Fresh cloud VM; the lab's managed Amazon session restored
  - [read] Product page read: full document title extends the short visible heading; page address, canonical address and Buy/Add form agree on the chosen kit, so one title alias is recorded
  - [cart] Exact chosen kit, quantity 1
  - [check] Checkout title matched the alias; quantity, seller, charges, requested address and requested card all matched
  - 38.8 s: [quote] Quote published: DJI Mic Mini (2 TX + 1 RX) $79.00 + tax $8.10 = $87.10
  - 49.1 s: [done] Session saved, VM closed
- What the record shows:
  - Items: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio × 1
  - Charges: $79.00 item + $8.10 tax = $87.10
  - Product identity: Accepted via source-bound title alias
  - Address and card: Requested ones matched
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Amazon · product_mismatch ×5 (Real · Oct 25, 2026, 8:21 PM PDT (ran 27 min 7 s; traced 9:03 PM PDT))

- Request: "can you get me the dji mic mini thing from amazon"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Live engine; price: $35.00 and $79.00 listings
- 384 trace events. The assistant asked Checkout Bench for quotes on two DJI Mic Mini kits. All five quotes were blocked as product_mismatch: the checkout showed a longer product title than the product page, and no product ID at checkout.
- Timeline:
  - [request] Person: “can you get me the dji mic mini thing from amazon”
  - [listing] Kit seen: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone, $35.00
  - [listing] Kit seen: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio, $79.00
  - [check] Checkout (2 TX + 1 RX): same wording plus extra descriptive text, 1 unit, $79 + $8.10 tax; no product ID readable
  - [block] Five quotes across the two kits: product_mismatch (exact-title rule)
  - [cleanup] All five cart restorations recorded
  - [reply] Assistant explained it as a same-product-ID issue and a transient error worth retrying; the record supports neither
- What the record shows:
  - Run time: 8:21:23 to 8:48:30 PM PDT (27 min 7 s)
  - Trace events: 384
  - Quotes: 5, all blocked as product_mismatch
  - Root cause: Short page title vs longer checkout title, no checkout product ID
  - Assistant's explanation: Went beyond the evidence
  - Cleanup: All five carts restored
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Instacart · product_choice_requires_user (Real · Oct 25, 2026 (start time not recorded, before 8:04 PM PDT))

- Request: "Regular Coca-Cola (exact message not recorded)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Live engine; price: No total
- Reached real sign-in and search, then stopped and asked the person to choose: the observed listing was a 24-pack and Checkout Bench could not tell it was the requested regular variant. No cart add, no purchase.
- Timeline:
  - [request] Request included the saved address and default card
  - [session] Real Instacart sign-in restored
  - [search] Search returned “Coca-Cola Soda (24 x 12 fl oz)”
  - [block] Quote blocked: product_choice_requires_user (fit to the requested regular variant unclear)
  - [reply] Assistant said the title did not confirm the amount, and read preferences.mode:none as “no saved address or card”; both went beyond the evidence
- What the record shows:
  - Run time: Not recorded
  - Cart add: Did not happen
  - Blocker: product_choice_requires_user
  - Assistant's explanation: Went beyond the evidence
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Amazon · Ready Checkout (Real · Oct 25, 2026, 7:55 PM PDT (ran 1 min 9 s))

- Request: "Crest toothpaste (exact message not recorded)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Live engine; price: $4.33
- Live quote run through the lab's new CLI/API. Complete quote with the account's selected address and card and no warnings.
- Timeline:
  - [start] Test started from the lab CLI/API: live, quote only
  - [quote] Checkout read: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
  - [check] Address and card: the account's selected ones, as requested
  - [quote] $3.93 item + $0.00 shipping + $0.40 tax = $4.33
  - [reply] Assistant reported the card the checkout actually showed instead of switching to its remembered preference
  - [cleanup] cleanup: done, no warnings; assistant VM stopped
- What the record shows:
  - Run time: 7:55:34 to 7:56:43 PM PDT (1 min 9 s)
  - Items: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
  - Charges: $3.93 + $0.00 shipping + $0.40 tax = $4.33
  - Address and card: Account-selected, as requested
  - Cleanup: Done, no warnings
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)

### Uber Eats · Delivered (read) (Real · Oct 25, 2026, 7:57 PM PDT (ran 47 s))

- Request: "Status of an earlier Uber Eats order (read-only continuation; exact message not recorded)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Live engine; price: n/a (read only)
- Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.
- Timeline:
  - [start] Read-only continuation of an earlier purchase
  - [status] One order-status request sent to Checkout Bench
  - 7:58:09 PM: [read] Order status: delivered (read from Uber's concluded-order view; scope: delivery status; receipt not re-checked)
  - [reply] Assistant kept the Oct 2 receipt and ETA separate from the fresh delivery reading and reported no current ETA
  - [done] Assistant VM stopped; zero quote or purchase requests
- What the record shows:
  - Run time: 7:57:32 to 7:58:19 PM PDT (47 s)
  - Status: delivered
  - Receipt re-checked: No
  - Quotes / purchases: 0 / 0
- Source: [Checkout Bench Research](https://www.checkoutbench.ai/research)


## Test flight (real, scrubbed)

15 cases × 2 models (DeepSeek V4.1 Flash, GPT-5.6 Luna) = 30 runs. Prompts were frozen before any run. One sample per case, shared real accounts, not a model ranking. Outcome labels were written by a reviewer, not graded by code.

| Case | Store | DeepSeek V4.1 Flash | GPT-5.6 Luna |
| --- | --- | --- | --- |
| A1 | Amazon | Assistant mistake: $4.33 proposal; usual tube unverified | Asked a fair question: Asked which usual tube |
| A2 | Amazon | Quoted correctly: $38.59 mobile kit; iPhone port unconfirmed | Quoted correctly: $38.59 mobile kit; iPhone port unconfirmed |
| A3 | Amazon | Checkout Bench backend bug: Matching 8-count pack refused | Checkout Bench backend bug: Matching 8-count pack refused |
| A4 | Amazon | Quoted correctly: $5.38; hard cap kept | Assistant mistake: $5.38; preferred card omitted |
| A5 | Amazon | Assistant mistake: $16.42; unsupported compatibility claim | Assistant mistake: $20.61; handle fit unconfirmed |
| A6 | Amazon | Assistant mistake: $88.85; soft target made hard | Checkout Bench backend bug: $29.97 item; longer-title block |
| U1 | Uber Eats | Asked a fair question: Asked which saved address | Assistant mistake: Searched an unverified default address |
| U2 | Uber Eats | Stopped by operator: Operator stopped after 5 attempts | Checkout Bench backend bug: Different options unsupported |
| U3 | Uber Eats | Stopped by operator: Operator stopped after 7 attempts | Assistant mistake: Menu block; unobserved alternative suggested |
| I1 | Instacart | Quoted correctly: $26.92; store discovery worked | Assistant mistake: Asked for a store without searching |
| I2 | Instacart | Checkout Bench backend bug: Same-store items; 12 vs 4 egg-count conflict | Assistant mistake: Asked for a store without searching |
| W1 | Walmart | Quoted correctly: $5.17 pickup; bag fee pending | Quoted correctly: $5.17 pickup; bag fee pending |
| W2 | Walmart | Quoted correctly: $3.26 pickup; 2 double rolls | Asked a fair question: Asked among observed pack choices |
| T1 | Target | Assistant mistake: 42-count unverified; shipping minimum unknown | Asked a fair question: 42-count unverified; shipping minimum unknown |
| C1 | Chewy | Assistant mistake: 30 lb bag unresolved; buy-once untested | Asked a fair question: 30 lb bag unresolved; buy-once untested |

### A1 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted)
- Group: Assistant mistake (reviewer label `partial`)
- Status: completed, 1 min 27.3 s
- What happened: Adult Crest gel, quantity one, $4.33 with reconciled charges and the saved destination and payment matched. The assistant presented it as a proposed gel and asked before going further, but never established that it was the usual white tube. The first request used unsupported options; the recovery dropped the variant constraint.
- Next step: Keep a literal product constraint in supported query or variant text after recovering from a schema error; ask about the usual product instead of assuming history.

### A1 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `appropriate_clarification`)
- Status: completed, 54.0 s
- What happened: Kept the white-tube query and quantity one. Checkout Bench returned product_choice_requires_user from sparse search titles; Luna showed the real alternatives, pointed out the multipacks and asked which tube is the usual one. No made-up history, cart change or purchase. An honest incomplete result, not proof that no white tube exists.
- Next step: None forced. Possible backend improvement: look at product-level evidence before escalating title-only uncertainty.

### A2 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one" Follow-up: "the phone one, 1 transmitter 1 receiver"
- Products picked: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone (ASIN `B0H94BWKB8`, quoted)
- Group: Quoted correctly (reviewer label `quote_with_compatibility_pending`)
- Status: completed, 2 min 9.6 s
- What happened: After the supplied follow-up, quoted the 1 TX + 1 Mobile RX kit at $38.59. The longer checkout title passed the released identity fix; quantity, charges, destination, payment and cart restore all verified. The reply said plainly that it is USB-C, but the shopper never said which iPhone port they have. The first Lightning query was not supported by the request.
- Next step: Settle USB-C vs Lightning before treating this as a fully compatible, ready-to-buy pick.

### A2 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one" Follow-up: "the phone one, 1 transmitter 1 receiver"
- Products picked: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone (ASIN `B0H94BWKB8`, quoted)
- Group: Quoted correctly (reviewer label `quote_with_compatibility_pending`)
- Status: completed, 1 min 6.9 s
- What happened: One accepted request picked the same phone kit at $38.59; full checkout title, quantity, charges and cleanup passed. The follow-up was never needed because no question was asked. The reply named USB-C and ruled out Lightning, but the actual iPhone port is still unknown.
- Next step: Keep it a conditional quote until the connector is confirmed. An accepted long title is not the same as a verified fit for the buyer's device.

### A3 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing"
- Products picked: none; not recorded
- Group: Checkout Bench backend bug (reviewer label `backend_false_mismatch`)
- Status: completed, 1 min 5.3 s
- What happened: Kept Duracell AA, 8-pack and quantity two. Checkout Bench returned product_choice_requires_user, calling an observed Duracell 8 Count listing a different pack size. The reply explained the conflict and proposed the exact 8-count listing twice (16 total), but asked for another approval it didn't need. No cart add, no purchase.
- Next step: Trace pack-size validation so per-listing pack count stays separate from order quantity. "8 pack" vs "8 Count" is the observed contradiction; the parser root cause isn't proven yet.

### A3 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing"
- Products picked: none; not recorded
- Group: Checkout Bench backend bug (reviewer label `backend_false_mismatch`)
- Status: completed, 57.5 s
- What happened: Same request and the same wrong 8-pack vs 8 Count refusal as DeepSeek. Luna found two matching 8-count Duracell packs in its reply but asked for another pick instead of reaching a quote. Saying Amazon needed a choice blamed the store for a Checkout Bench validation stop.
- Next step: Fix the shared pack validation rather than patching two model prompts; quantity two was read correctly.

### A4 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls"
- Products picked: Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 Foot (ASIN `B085SBNFNP`, quoted)
- Group: Quoted correctly (reviewer label `correct_quote`)
- Status: completed, 1 min 1.2 s
- What happened: One 6 ft USB-C to USB-C cable: $4.88 + $0 shipping + $0.50 tax = $5.38, under the $12 hard cap. Exact connector, length and full total; saved destination and payment matched.
- Next step: None for this run.

### A4 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls"
- Products picked: Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 Foot (ASIN `B085SBNFNP`, quoted)
- Group: Assistant mistake (reviewer label `correct_product_budget_with_payment_omission`)
- Status: completed, 1 min 2.0 s
- What happened: Same correct cable and $5.38 total, under the $12 cap. But the shopper context names a preferred card for Amazon; Luna left payment out of its request, so the account's default card came back unmatched. The reply said Amazon didn't match the preferred card, though no such selection was ever requested or tested.
- Next step: Carry known preferences into the request, and tell an account default apart from a tested payment mismatch. An assistant omission, not evidence the store rejected the card.

### A5 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted); Oral-B Deep Clean Replacement Brush Heads, 2 Count (ASIN `B003UBTYKI`, quoted)
- Group: Assistant mistake (reviewer label `unsupported_compatibility_claim`)
- Status: completed, 1 min 41.4 s
- What happened: Both items in one reconciled $16.42 basket, cleanup done. The heads were Oral-B Deep Clean battery-brush replacements; the buyer's handle type was unknown. The reply claimed they are standard refills for rechargeable handles with no data to back it, and the request dropped the "regular electric brush" detail.
- Next step: Keep compatibility constraints through search and find out the handle type, or ask. A ready checkout alone doesn't approve the pick.

### A5 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted); Oral-B Daily Clean Replacement Brush Heads, Pack of 3 (ASIN `B089LBBDZR`, quoted)
- Group: Assistant mistake (reviewer label `compatibility_unconfirmed`)
- Status: completed, 1 min 21.7 s
- What happened: Kept the regular electric-brush detail and chose Daily Clean 3-pack heads with the Crest in one $20.61 basket, cleanup done. Still didn't establish the handle type or caveat fit. The reply labeled a 10:15 PM local expiry as UTC (it was 05:15 UTC). One refused out-of-scope read is in the record; the quote itself worked.
- Next step: Confirm the handle type or qualify the pick; fix the time-zone wording separately.

### A6 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50"
- Products picked: STANLEY Quencher H2.0 FlowState Tumbler (Peony, 40 oz) (ASIN `B0CZF97C8N`, quoted)
- Group: Assistant mistake (reviewer label `soft_budget_misclassified`)
- Status: completed, 1 min 37.8 s
- What happened: Pink Peony Stanley 40 oz, full quote $88.85, above target; arrival Oct 27, before Saturday Oct 31; seller, charges, destination and payment shown; cart restore verified. The assistant sent the budget as a hard $50 cap, unlike the tester's soft-target note, stopped after one search and asked about other pinks though any pink was allowed.
- Next step: Keep target vs hard cap apart from store over-budget behavior; search within budget before asking for a preference already given. The wording is somewhat ambiguous, so one run doesn't prove a general budget bug.

### A6 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50"
- Products picked: STANLEY Quencher H2.0 Flow State Tumbler, 40 oz, Peony (ASIN `B0CRMP3RQT`, selected; Checkout Bench blocked on title)
- Group: Checkout Bench backend bug (reviewer label `identity_block_and_reply_gaps`)
- Status: completed, 1 min 10.9 s
- What happened: Saw a pink Peony 40 oz at $29.97 item price, but Checkout Bench blocked it because the checkout title was longer than the selected title at the same price. The released DJI fix passes; this different title case is still open. No full charges or arrival shown. The reply called it under $50 from the item price, left out Saturday, and treated the soft target as hard. Cart restore verified. Also: Reply also called it under budget from item price alone and dropped the Saturday deadline.
- Next step: Read the retained product-identity evidence for this title block before changing the identity rule again; keep the deadline and separate item price from delivered total.

### U1 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `appropriate_address_clarification`)
- Status: completed, 53.7 s
- What happened: Kept the Jumbo Jack meal, medium Coke and tip. The saved address label in the request didn't match the label on the account, and the assistant asked which saved address to use, with no made-up answer. No menu, price, tip or quote proved; no cart change.
- Next step: A durable buyer alias or evidence-based address match could avoid this question; don't guess a destination from unrelated store context.

### U1 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `unverified_home_and_store_unavailable`)
- Status: completed, 31.5 s
- What happened: Kept the meal, medium Coke and tip but left out the destination. Checkout Bench searched the account default and returned store_not_found. The reply said that was the requested saved address without evidence the labels matched; nearby results don't prove availability at the intended address. No quote, tip or cart change.
- Next step: Resolve the intended saved address before blaming store availability on it; don't switch restaurant or address without asking.

### U2 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it"
- Products picked: none; not recorded
- Group: Stopped by operator (reviewer label `unsupported_configuration_and_retry_loop_operator_stopped`)
- Status: cancelled, 4 min 43.4 s
- What happened: The first request correctly listed the two large fries as separate configurations, one no-salt. Checkout Bench refuses identical items with different options (duplicate_product). The assistant then tried four single no-salt requests; all returned item_dialog_missing, so no-salt availability stayed unknown. The operator stopped the loop after five accepted quotes; no final reply. Later reads confirmed all five blocked and settled, no cart adds, VM stopped. Also: Checkout Bench can't hold two of the same item with different options (duplicate_product).
- Next step: Separate configured line identity from product identity, and stop unchanged retries after a repeated reader failure. Not an unassisted finish.

### U2 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it"
- Products picked: none; not recorded
- Group: Checkout Bench backend bug (reviewer label `honest_unsupported_with_attribution_gap`)
- Status: completed, 43.0 s
- What happened: Kept two distinct salt configurations, hit the same duplicate_product limit and stopped after one request without merging or dropping the restriction. The reply blamed Uber Eats rather than Checkout Bench and offered two no-salt fries without evidence that option exists. No cart adds or quote. Also: Reply blamed Uber Eats instead of Checkout Bench and offered an unverified alternative.
- Next step: Explain Checkout Bench's limit accurately and label unverified alternatives as such; support configuration-aware line identity in the backend.

### U3 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one"
- Products picked: none; not recorded
- Group: Stopped by operator (reviewer label `repeated_menu_store_failures_operator_stopped`)
- Status: cancelled, 5 min 2.9 s
- What happened: The first request kept both large pizzas, but menu matching failed. Seven accepted quotes alternated product_choice_requires_user and store_not_found as the query and restaurant name changed. Recovery dropped the sizes for a while and once shrank to "pizza, quantity two", losing the toppings. The operator stopped the loop; no final reply, basket or deal. Later reads confirmed all seven blocked and settled, no cart steps, VM stopped. Also: Menu matching failed repeatedly on Checkout Bench's side.
- Next step: Show the actual observed menu and availability evidence and keep the original constraints while probing; stop repeated failures without inventing a pizza or deal.

### U3 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `menu_match_block_with_unverified_alternative`)
- Status: completed, 46.9 s
- What happened: Kept both large pizzas, then stopped after one product_choice_requires_user refusal for the pepperoni. No cheese search, deal or checkout. The reply admitted the quote was incomplete but proposed a specifically named Large 1-Topping Pizza that wasn't in what Checkout Bench returned. No cart activity. Also: Checkout Bench also stopped on a menu match for the pepperoni.
- Next step: Return observed menu candidates and keep going on independent items instead of inventing a plausible menu option.

### I1 · Instacart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you"
- Products picked: Coca-Cola Soda, 24 x 12 fl oz (Instacart product `93587`, quoted)
- Group: Quoted correctly (reviewer label `correct_quote_cheapest_delivered_unproven`)
- Status: completed, 1 min 52.1 s
- What happened: Found local offers, chose Walmart's 24 x 12 fl oz regular Coke cans ($15.52 merchandise) and kept the product and store link through the follow-up. Reconciled checkout $26.92 including service, long-distance, tax and fee lines plus tip. Walmart had the lowest matching product price, but other stores' full fees weren't quoted, so cheapest delivered total is unproven.
- Next step: Keep product-price comparison separate from delivered-total comparison.

### I1 · Instacart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `unnecessary_retailer_clarification`)
- Status: completed, 9.1 s
- What happened: Sent no search or quote. Asked the shopper to name a retailer and said Instacart needs one, though the single-item store-discovery path was available (DeepSeek used it in the previous run). Misses the prewritten outcome of finding the cheapest store. No cart activity. DeepSeek read the Instacart-specific note; Luna read only the general files.
- Next step: Line up guide use with what the tool can actually do; ask only for missing preferences. Check what the assistant read before blaming only the model.

### I2 · Instacart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees"
- Products picked: Marketside Grade A Cage Free Large Brown Eggs, 12 ct (Instacart product `20659148`, selected; blocked before cart); Great Value 2% Reduced Fat Milk, 128 oz (Instacart product `20630136`, selected; blocked before cart)
- Group: Checkout Bench backend bug (reviewer label `conflicting_pack_evidence_unresolved`)
- Status: completed, 1 min 55.0 s
- What happened: Found both items at one Walmart and kept exact product links for the follow-up. The egg offer says 12 ct, the milk 128 oz. The follow-up called the same egg listing a pack of four and blocked before adding to cart, so the real pack size is unresolved. The assistant repeated "four" as fact and proposed quantity three. Milk found in search only, not verified at checkout; no basket or fees. Also: The assistant repeated the disputed pack count as fact.
- Next step: Trace pack-count reading against the retained source evidence before suggesting three packs; tell a candidate basket apart from a verified checkout.

### I2 · Instacart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `premature_retailer_clarification`)
- Status: completed, 10.7 s
- What happened: Asked which store to use for both items without any search or quote attempt, though same-store discovery was available (DeepSeek used it in the previous run). Nothing dropped or falsely quoted, but an extra interruption before trying the tool.
- Next step: Use store discovery before turning a tool-solvable store choice into a required question; an optional preference can be offered without blocking.

### W1 · Walmart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway"
- Products picked: Great Value Purified Drinking Water, 16.9 fl oz, 24 Count (Walmart item `13448854`, quoted)
- Group: Quoted correctly (reviewer label `correct_pickup_quote_pending_bag_fee`)
- Status: completed, 1 min 1.6 s
- What happened: One Great Value 24-count water pack, pickup as requested. Reconciled quote $5.17 ($3.97 items + $1.20 taxes and fees). The reply disclosed the pending bag fee and listed a $5.46 authorization-hold amount separately. A pickup quote, not a booking or a placed hold.
- Next step: Keep pending fees and a possible future hold separate from the quote and any real charge.

### W1 · Walmart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway"
- Products picked: Great Value Purified Drinking Water, 16.9 fl oz, 24 Count (Walmart item `13448854`, quoted)
- Group: Quoted correctly (reviewer label `correct_pickup_quote_pending_bag_fee`)
- Status: completed, 1 min 7.5 s
- What happened: The same Great Value 24-count water pack for pickup, $5.17 reconciled estimate. The reply disclosed the pending bag fee and called it a quote; no shipping swap or booking claim.
- Next step: No correction needed; keep the pending-fee caveat before buying.

### W2 · Walmart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup"
- Products picked: Great Value Everyday Strong Paper Towels, 2 Double Rolls (Walmart item `10762707702`, quoted)
- Group: Quoted correctly (reviewer label `brand_pickup_correct_pack_ambiguous`)
- Status: completed, 1 min 0.6 s
- What happened: Switched Bounty to Great Value and kept pickup. Chose one 2 Double Rolls package at $3.26 with tax, with the pending bag fee disclosed. The request dropped the original "big pack" wording, so the intended size is unclear; no made-up size.
- Next step: Keep a soft pack-size wish while changing brands and show the pack chosen; don't invent a number from "big pack".

### W2 · Walmart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `observed_pack_clarification`)
- Status: completed, 46.9 s
- What happened: Kept the switch to Great Value, pickup and the "big pack" wish. Checkout Bench found 12 triple, 6 roll and 2 roll choices but needed an exact pack; Luna asked among those real options, with no made-up size or shipping fallback. No cart activity. Meets the prewritten allowance for a pack question.
- Next step: Consider picking from observed packs for soft size wishes instead of treating a vague size as exact.

### T1 · Target · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `honest_search_block_unverified_suggestions`)
- Status: completed, 1 min 38.8 s
- What happened: Two requests kept Tide PODS 42-count and shipping; both returned product_choice_requires_user with no candidates. The assistant correctly said search failed before cart, so minimum, fees and total are unknown, with no filler or pack change. But it added unverified claims about Target's usual pack counts and unneeded internal preference details. No cart activity. Also: Checkout Bench also returned no candidate listings.
- Next step: Return observed listings or source-backed reasons for no match; avoid catalog guesses.

### T1 · Target · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `honest_product_match_block`)
- Status: completed, 59.5 s
- What happened: One request kept Tide PODS 42-count and shipping. Checkout Bench found no fitting listing and returned no alternatives. The assistant said no price or minimum was verified, didn't claim it was out of stock, and asked before changing pack or quantity. No cart activity.
- Next step: Show the candidates Checkout Bench inspected and why they didn't match.

### C1 · Chewy · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `exact_bag_unresolved_one_time_unexercised`)
- Status: completed, 2 min 49.5 s
- What happened: Two attempts kept Chicken & Rice and 30 lb; the second added Complete Essentials as a search refinement. Both blocked on product match, showing senior 7+ or other flavors. The existing cart was set aside and restored both times; no cart add or Autoship. The reply asked before substituting but wrongly said the shopper named Complete Essentials, and described the buy-once policy without reaching that checkout step. Also: Checkout Bench blocked both attempts on product match.
- Next step: Resolve the exact formula and weight from observed choices or a product link; don't credit the shopper with an inferred product line.

### C1 · Chewy · GPT-5.6 Luna

- Real prompt, scrubbed (frozen before the runs; hash checked for both models): "dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `exact_bag_unresolved_one_time_unexercised`)
- Status: completed, 1 min 9.6 s
- What happened: One request kept the exact adult Chicken & Rice 30 lb target. Checkout Bench showed senior 7+ or other flavors but no exact bag. The assistant left it unresolved and asked before changing anything, without claiming a quote or buy-once check. Existing cart set aside and restored; no cart add or Autoship.
- Next step: Coverage gap: a verified exact product is still needed to test one-time checkout.

## Task bank

| ID | Store | Shopper typed | Counts as correct | Difficulty |
|---|---|---|---|---|
| T-001 | Amazon | can you get me the dji mic mini thing from amazon | Any DJI Mic Mini kit after asking which kit | hard |
| T-014 | Walmart | need paper towels the big pack, bounty not the off brand walmart pls | Bounty Select-A-Size, 12 double rolls or larger | medium |
| T-022 | Chewy | dog food same as last time?? the purina pro plan salmon one, 30lb | Purina Pro Plan Adult Salmon & Rice, 30 lb | easy |
| T-031 | Target | get 2 of those owala water bottles 24oz, the blue-ish one my kid likes | Owala FreeSip 24 oz, any blue colorway, qty 2 | medium |
| T-040 | Instacart | bananas (not too green lol), 2 gal whole milk, eggs 18ct, whatever bread is cheapest | 4 items; cheapest sandwich loaf at that store | hard |
| T-047 | Uber Eats | chipotle bowl chicken double rice no beans, under 20 bucks w/ fees | Chicken bowl, extra rice, no beans; total ≤ $20 | hard |
| T-055 | Amazon | my airpods case died can u find just the charging case, gen 2 usb c | Apple charging case only, USB-C; ask if no case-only listing exists | hard |
| T-063 | Target | tide pods free & clear like 80ish count | Tide PODS Free & Clear, 72–91 count | medium |
| T-070 | Amazon | ink for my HP envy 6055e, black AND color | HP 67 or 67XL black + tri-color | medium |
| T-078 | Chewy | cat litter that doesnt smell, unscented clumping, 40 lb-ish, under $30 | Unscented clumping, 35–42 lb, ≤ $30 | medium |
| T-086 | Target | huggies sensitive wipes, 3 packs, whatever count is the better deal | Huggies Sensitive, lowest price per wipe, 3 packs | hard |
| T-093 | Amazon | uh the ninja air fryer thats like 100 bucks, 4qt? | Ninja 4 qt air fryer, $80–120; ask if several match | medium |