---
title: "Test suite · Checkout Bench"
description: "Checkout Bench test suite: recent real Nook runs with condensed timelines (scrubbed), plus a leaderboard and task bank that are still example data."
canonical: https://www.checkoutbench.ai/suite
last_updated: 2026-10-25
---

# Test suite

> Recent runs are real (scrubbed). Leaderboard and task bank are example data: model names, pass rates and grades are made up to show the layout.

## Leaderboard (example data)

| Model | Amazon | Walmart | Target | Instacart | Uber Eats | Chewy |
|---|---|---|---|---|---|---|
| Model A | 71% | 64% | 58% | 49% | 66% | 62% |
| Model B | 63% | 59% | 61% | 40% | 52% | 57% |
| Model C | 55% | 48% | 44% | 37% | 50% | 46% |
| Model D | 41% | 39% | 33% | 22% | 35% | 30% |
| Model E | 29% | 31% | 24% | 18% | 27% | 21% |
| Model F | 12% | 15% | 9% | 6% | 14% | 10% |

## Charts (real)

Source: Nook quote lab run records (458 runs, Sep 26 to Oct 25, 2026 PDT), aggregated read-only; Nook CHANGELOG.md at main 8ca92315.

- Lab runs per day (PDT): Sep 26 76, Sep 27 39, Sep 28 12, Sep 29 25, Oct 1 264, Oct 2 13, Oct 25 29
- Run outcomes (596 assistant runs): Quote ready 168, Quote blocked 240, Asked the person 95, No quote 62, Other 31
- Quotes by store (ready/blocked/preparing): Amazon 101/87/0, Uber Eats 34/95/2, Instacart 23/78/1, Walmart 13/41/2, Chewy 10/17/0, Target 5/18/1
- Top blockers: product_choice_requires_user 140, product_mismatch 19, checkout_link_missing 12, item_dialog_missing 12, below_store_minimum 12, store_not_found 12, item_choices_required 11, quote_failed 10, over_budget 10, checkout_not_ready 9
- Seconds to a quote (median, p90): Walmart 34.3/74.5, Amazon 34.6/60.7, Instacart 39.3/49.6, Uber Eats 46.2/117.8, Target 56.8/115.7, Chewy 63.1/90.3
- Median seconds to stage: Browser ready 8.2, Signed in 16.2, Search results 23.3, Product chosen 26.4, Checkout read 39.7, Quote checked 43.9, Cart restored 49.2
- Assistant models (runs, median events, median seconds, median model cost USD): GPT-5.6 Sol 381, 77, 47.2, 0.0639; DeepSeek V4.1 Flash 129, 95, 64.0, 0.0068; GPT-5.6 Luna 51, 105, 78.9, 0.0047; Claude Opus 4.8 10, 85.5, 64.4, 0.0924; Grok 4.7 10, 82, 56.0, 0.0116; Kimi K3 9, 84, 58.4, 0.0295
- Ready quote totals (median, min to max USD): Amazon 11.0 (4.33 to 220.5), Uber Eats 26.8 (15.22 to 44.91), Instacart 26.8 (16.97 to 46.29), Walmart 13.1 (4.88 to 21.31), Chewy 9.3 (8.25 to 40.06), Target 40.5 (8.58 to 40.53)
- Listing vs quoted: DJI Mic Mini (2 TX + 1 RX) 79.00 to 87.10, Crest Pro-Health toothpaste 4.3 oz 3.93 to 4.33
- Tests passing (Oct 25, 9:13 PM PDT): Cloud 436/436, Lab 170/170, Service 140/140, Onboarding 18/18
- Changelog entries per day: Sep 17 29, Sep 18 18, Sep 22 14, Sep 23 38, Sep 25 3, Sep 26 28, Sep 27 42, Sep 28 28, Sep 29 13, Oct 2 24, Oct 25 8
- Browser replays: placeholders only (16:9, 9:16, 4:3); runs keep event logs, not page recordings.


## Recent runs (real, Oct 25 to 26, 2026, scrubbed)

### Amazon · Reached order review (Real · Oct 26, 2026, 4:12 PM PDT (ran 41.2 s))

- Request: "USB-C to USB-C charging cable, 6 ft, quantity 1, under $15"
- Assistant: typesafe-ai/jev; provider: Not recorded; Nook v3 Amazon quote driver (signed-in cloud Chrome); price: $5.38
- Signed-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. It stopped there; nothing was bought. Score 1.0 (4 of 4 checks passed).
- Timeline:
  - 0 s: [start] Signed-in cloud Chrome session
  - [search] Right item found: Amazon Basics USB-C to USB-C cable, 6 ft
  - [check] Reached order review: $5.38 total vs $15 cap
  - 41.2 s: [done] Stopped before payment; nothing bought
- What the record shows:
  - Items: Amazon Basics USB-C to USB-C cable, 6 ft × 1
  - Right item: Pass
  - Reached review: Pass
  - Under price cap: $5.38 vs $15
  - Stopped before payment: Pass
  - Purchases: 0
- Source: Run record 2026-10-26-amazon-nook-v3-01 (result-entry.json)

### Best Buy · Found product, no cart or checkout (Real · Oct 26, 2026, 4:45 PM PDT (ran 35.9 s))

- Request: "USB-C to USB-C charging cable, 6 ft, quantity 1, under $15"
- Assistant: No assistant (direct tool calls); provider: n/a; Pomerado Best Buy tool (search + product details), called directly; price: $14.95 before tax
- Signed-out run that found a 6 ft USB-C to USB-C cable listed at $14.95 before tax, under the $15 cap, with free shipping by Oct 30. The Pomerado tool has no cart step, so nothing was added to cart, no order total was seen, and nothing was bought. The checks below cover only what a product page can show.
- Timeline:
  - 0 s: [start] Pomerado Best Buy tool, signed out, shipping ZIP set
  - 20.0 s: [search] 20 listings; right item found: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, $14.95
  - 35.9 s: [read] Product details: SKU 10555185, $14.95, Shipping FREE Fri, Oct 30; pickup not listed
  - 35.9 s: [done] Stopped at product quote; no cart step in this tool; nothing bought
- What the record shows:
  - Items: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, Black (SKU 10555185) × 1
  - Right item: Pass
  - Under price cap: $14.95 vs $15 (item price, before tax)
  - In stock or pickup: Free shipping, arrives Fri, Oct 30; pickup not listed
  - Stopped before cart: Pass
  - Reached review: No: the tool has no cart or checkout step
  - Purchases: 0
- Source: Run record 2026-10-26-bestbuy-pomerado-01 (result-entry.json)

### Amazon · Ready quote (Real · Oct 25, 2026, 9:12 PM PDT)

- Request: "Re-run of the last DJI request from the run below, on the exact kit the person had chosen"
- Assistant: No assistant (direct engine check); provider: n/a; Candidate engine (PR #158), not production; price: $87.10
- The same DJI Mic Mini (2 TX + 1 RX) that production had blocked as product_mismatch, re-quoted with the PR #158 title-alias fix. Quote-only; nothing bought.
- Timeline:
  - 0 s: [start] Fresh cloud VM; the lab's managed Amazon session restored
  - [read] Product page read: full document title extends the short visible heading; page address, canonical address and Buy/Add form agree on the chosen kit, so one title alias is recorded
  - [cart] Exact chosen kit, quantity 1
  - [check] Checkout title matched the alias; quantity, seller, charges, requested address and requested card all matched
  - 38.8 s: [quote] Quote published: DJI Mic Mini (2 TX + 1 RX) $79.00 + tax $8.10 = $87.10
  - 47.4 s: [cleanup] Original empty cart restored; bought: 0
  - 49.1 s: [done] Session saved, VM closed
- What the record shows:
  - Items: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio × 1
  - Charges: $79.00 item + $8.10 tax = $87.10
  - Product identity: Accepted via source-bound title alias
  - Address and card: Requested ones matched
  - Cleanup: Done, cart restored
  - Purchases: 0
- Source: nook-v3/CHANGELOG.md 2026-10-25 21:12; nook-v3/cloud-connect/AMAZON-IDENTITY.md

### Amazon · product_mismatch ×5 (Real · Oct 25, 2026, 8:21 PM PDT (ran 27 min 7 s; traced 9:03 PM PDT))

- Request: "can you get me the dji mic mini thing from amazon"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Production engine; price: $35.00 and $79.00 listings
- 384 trace events. The assistant asked Nook for quotes on two DJI Mic Mini kits. All five quotes were blocked as product_mismatch: the checkout showed a longer product title than the product page, and no product ID at checkout. Nothing bought.
- Timeline:
  - [request] Person: “can you get me the dji mic mini thing from amazon”
  - [listing] Kit seen: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone, $35.00
  - [listing] Kit seen: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio, $79.00
  - [check] Checkout (2 TX + 1 RX): same wording plus extra descriptive text, 1 unit, $79 + $8.10 tax; no product ID readable
  - [block] Five quotes across the two kits: product_mismatch (exact-title rule)
  - [cleanup] All five cart restorations recorded
  - [reply] Assistant explained it as a same-product-ID issue and a transient error worth retrying; the record supports neither
- What the record shows:
  - Run time: 8:21:23 to 8:48:30 PM PDT (27 min 7 s)
  - Trace events: 384
  - Quotes: 5, all blocked as product_mismatch
  - Root cause: Short page title vs longer checkout title, no checkout product ID
  - Assistant's explanation: Went beyond the evidence
  - Cleanup: All five carts restored
  - Purchases: 0
- Source: Run summary shared in chat (model, request); lab run record (start, finish, provider, event count); nook-v3/CHANGELOG.md 2026-10-25 21:03; nook-v3/cloud-connect/AMAZON-IDENTITY.md

### Instacart · product_choice_requires_user (Real · Oct 25, 2026 (start time not recorded, before 8:04 PM PDT))

- Request: "Regular Coca-Cola (exact message not in the repo record)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Production engine; price: No total
- Reached real sign-in and search, then stopped and asked the person to choose: the observed listing was a 24-pack and Nook could not tell it was the requested regular variant. No cart add, no purchase.
- Timeline:
  - [request] Request included the saved address and default card
  - [session] Real Instacart sign-in restored
  - [search] Search returned “Coca-Cola Soda (24 x 12 fl oz)”
  - [block] Quote blocked: product_choice_requires_user (fit to the requested regular variant unclear)
  - [reply] Assistant said the title did not confirm the amount, and read preferences.mode:none as “no saved address or card”; both went beyond the evidence
- What the record shows:
  - Run time: Not recorded
  - Cart add: Did not happen
  - Blocker: product_choice_requires_user
  - Assistant's explanation: Went beyond the evidence
  - Purchases: 0
- Source: nook-v2/lab/docs/MAIN-INTEGRATION.md (Live evidence and limits); nook-v3/CHANGELOG.md 2026-10-25 20:04

### Amazon · Ready quote (Real · Oct 25, 2026, 7:55 PM PDT (ran 1 min 9 s))

- Request: "Crest toothpaste (exact message not in the repo record)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Production engine; price: $4.33
- Quote-only production run through the lab's new CLI/API. Complete quote with the account's selected address and card, cart cleaned up, no warnings.
- Timeline:
  - [start] Test started from the lab CLI/API: production, quote only
  - [quote] Checkout read: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
  - [check] Address and card: the account's selected ones, as requested
  - [quote] $3.93 item + $0.00 shipping + $0.40 tax = $4.33
  - [reply] Assistant reported the card the checkout actually showed instead of switching to its remembered preference
  - [cleanup] cleanup: done, no warnings; assistant VM stopped
- What the record shows:
  - Run time: 7:55:34 to 7:56:43 PM PDT (1 min 9 s)
  - Items: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
  - Charges: $3.93 + $0.00 shipping + $0.40 tax = $4.33
  - Address and card: Account-selected, as requested
  - Cleanup: Done, no warnings
  - Purchases: 0
- Source: nook-v3/CHANGELOG.md 2026-10-25 19:59 and 20:04; nook-v2/lab/docs/MAIN-INTEGRATION.md; lab run record (start, finish)

### Uber Eats · Delivered (read) (Real · Oct 25, 2026, 7:57 PM PDT (ran 47 s))

- Request: "Status of an earlier Uber Eats order (read-only continuation; exact message not in the repo record)"
- Assistant: DeepSeek V4.1 Flash; provider: Gateway; Production engine; price: n/a (read only)
- Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.
- Timeline:
  - [start] Read-only continuation of an earlier purchase
  - [status] One order-status request sent to Nook
  - 7:58:09 PM: [read] Order status: delivered (read from Uber's concluded-order view; scope: delivery status; receipt not re-checked)
  - [reply] Assistant kept the Oct 2 receipt and ETA separate from the fresh delivery reading and reported no current ETA
  - [done] Assistant VM stopped; zero quote or purchase requests
- What the record shows:
  - Run time: 7:57:32 to 7:58:19 PM PDT (47 s)
  - Status: delivered
  - Receipt re-checked: No
  - Quotes / purchases: 0 / 0
- Source: nook-v3/CHANGELOG.md 2026-10-25 19:59 and 20:04; nook-v2/lab/docs/MAIN-INTEGRATION.md; lab run record (start, finish)


## Test flight (real, Oct 25, 2026, scrubbed)

15 cases × 2 models (DeepSeek V4.1 Flash, GPT-5.6 Luna) = 30 runs. Source commit 8ca92315; prompts frozen at c3d38e2. No purchases. One sample per case, shared real accounts, not a model ranking. Outcome labels were written by a reviewer, not graded by code.

| Case | Store | DeepSeek V4.1 Flash | GPT-5.6 Luna |
| --- | --- | --- | --- |
| A1 | Amazon | Assistant mistake: $4.33 proposal; usual tube unverified | Asked a fair question: Asked which usual tube |
| A2 | Amazon | Quoted correctly: $38.59 mobile kit; iPhone port unconfirmed | Quoted correctly: $38.59 mobile kit; iPhone port unconfirmed |
| A3 | Amazon | Nook backend bug: Matching 8-count pack refused | Nook backend bug: Matching 8-count pack refused |
| A4 | Amazon | Quoted correctly: $5.38; hard cap kept | Assistant mistake: $5.38; preferred card omitted |
| A5 | Amazon | Assistant mistake: $16.42; unsupported compatibility claim | Assistant mistake: $20.61; handle fit unconfirmed |
| A6 | Amazon | Assistant mistake: $88.85; soft target made hard | Nook backend bug: $29.97 item; longer-title block |
| U1 | Uber Eats | Asked a fair question: Asked which saved address | Assistant mistake: Searched an unverified default address |
| U2 | Uber Eats | Stopped by operator: Operator stopped after 5 attempts | Nook backend bug: Different options unsupported |
| U3 | Uber Eats | Stopped by operator: Operator stopped after 7 attempts | Assistant mistake: Menu block; unobserved alternative suggested |
| I1 | Instacart | Quoted correctly: $26.92; store discovery worked | Assistant mistake: Asked for a store without searching |
| I2 | Instacart | Nook backend bug: Same-store items; 12 vs 4 egg-count conflict | Assistant mistake: Asked for a store without searching |
| W1 | Walmart | Quoted correctly: $5.17 pickup; bag fee pending | Quoted correctly: $5.17 pickup; bag fee pending |
| W2 | Walmart | Quoted correctly: $3.26 pickup; 2 double rolls | Asked a fair question: Asked among observed pack choices |
| T1 | Target | Assistant mistake: 42-count unverified; shipping minimum unknown | Asked a fair question: 42-count unverified; shipping minimum unknown |
| C1 | Chewy | Assistant mistake: 30 lb bag unresolved; buy-once untested | Asked a fair question: 30 lb bag unresolved; buy-once untested |

### A1 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted)
- Group: Assistant mistake (reviewer label `partial`)
- Status: completed, 1 min 27.3 s
- What happened: Adult Crest gel, quantity one, $4.33 with reconciled charges and the saved destination and payment matched. The assistant presented it as a proposed gel and asked before going further, but never established that it was the usual white tube. The first request used unsupported options; the recovery dropped the variant constraint.
- Next step: Keep a literal product constraint in supported query or variant text after recovering from a schema error; ask about the usual product instead of assuming history.

### A1 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `appropriate_clarification`)
- Status: completed, 54.0 s
- What happened: Kept the white-tube query and quantity one. Nook returned product_choice_requires_user from sparse search titles; Luna showed the real alternatives, pointed out the multipacks and asked which tube is the usual one. No made-up history, cart change or purchase. An honest incomplete result, not proof that no white tube exists.
- Next step: None forced. Possible backend improvement: look at product-level evidence before escalating title-only uncertainty.

### A2 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one" Follow-up: "the phone one, 1 transmitter 1 receiver"
- Products picked: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone (ASIN `B0H94BWKB8`, quoted)
- Group: Quoted correctly (reviewer label `quote_with_compatibility_pending`)
- Status: completed, 2 min 9.6 s
- What happened: After the supplied follow-up, quoted the 1 TX + 1 Mobile RX kit at $38.59. The longer checkout title passed the released identity fix; quantity, charges, destination, payment and cart restore all verified. The reply said plainly that it is USB-C, but the shopper never said which iPhone port they have. The first Lightning query was not supported by the request.
- Next step: Settle USB-C vs Lightning before treating this as a fully compatible, ready-to-buy pick.

### A2 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one" Follow-up: "the phone one, 1 transmitter 1 receiver"
- Products picked: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone (ASIN `B0H94BWKB8`, quoted)
- Group: Quoted correctly (reviewer label `quote_with_compatibility_pending`)
- Status: completed, 1 min 6.9 s
- What happened: One accepted request picked the same phone kit at $38.59; full checkout title, quantity, charges and cleanup passed. The follow-up was never needed because no question was asked. The reply named USB-C and ruled out Lightning, but the actual iPhone port is still unknown.
- Next step: Keep it a conditional quote until the connector is confirmed. An accepted long title is not the same as a verified fit for the buyer's device.

### A3 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing"
- Products picked: none; not recorded
- Group: Nook backend bug (reviewer label `backend_false_mismatch`)
- Status: completed, 1 min 5.3 s
- What happened: Kept Duracell AA, 8-pack and quantity two. Nook returned product_choice_requires_user, calling an observed Duracell 8 Count listing a different pack size. The reply explained the conflict and proposed the exact 8-count listing twice (16 total), but asked for another approval it didn't need. No cart add, no purchase.
- Next step: Trace pack-size validation so per-listing pack count stays separate from order quantity. "8 pack" vs "8 Count" is the observed contradiction; the parser root cause isn't proven yet.

### A3 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing"
- Products picked: none; not recorded
- Group: Nook backend bug (reviewer label `backend_false_mismatch`)
- Status: completed, 57.5 s
- What happened: Same request and the same wrong 8-pack vs 8 Count refusal as DeepSeek. Luna found two matching 8-count Duracell packs in its reply but asked for another pick instead of reaching a quote. Saying Amazon needed a choice blamed the store for a Nook validation stop.
- Next step: Fix the shared pack validation rather than patching two model prompts; quantity two was read correctly.

### A4 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls"
- Products picked: Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 Foot (ASIN `B085SBNFNP`, quoted)
- Group: Quoted correctly (reviewer label `correct_quote`)
- Status: completed, 1 min 1.2 s
- What happened: One 6 ft USB-C to USB-C cable: $4.88 + $0 shipping + $0.50 tax = $5.38, under the $12 hard cap. Exact connector, length and full total; saved destination and payment matched; cart restored.
- Next step: None for this run.

### A4 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls"
- Products picked: Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 Foot (ASIN `B085SBNFNP`, quoted)
- Group: Assistant mistake (reviewer label `correct_product_budget_with_payment_omission`)
- Status: completed, 1 min 2.0 s
- What happened: Same correct cable and $5.38 total, under the $12 cap, cart cleaned up. But the shopper context names a preferred card for Amazon; Luna left payment out of its request, so the account's default card came back unmatched. The reply said Amazon didn't match the preferred card, though no such selection was ever requested or tested.
- Next step: Carry known preferences into the request, and tell an account default apart from a tested payment mismatch. An assistant omission, not evidence the store rejected the card.

### A5 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted); Oral-B Deep Clean Replacement Brush Heads, 2 Count (ASIN `B003UBTYKI`, quoted)
- Group: Assistant mistake (reviewer label `unsupported_compatibility_claim`)
- Status: completed, 1 min 41.4 s
- What happened: Both items in one reconciled $16.42 basket, cleanup done. The heads were Oral-B Deep Clean battery-brush replacements; the buyer's handle type was unknown. The reply claimed they are standard refills for rechargeable handles with no data to back it, and the request dropped the "regular electric brush" detail.
- Next step: Keep compatibility constraints through search and find out the handle type, or ask. A ready quote alone doesn't approve the pick.

### A5 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together"
- Products picked: Crest Pro-Health Whitening Gel Toothpaste (4.3oz) (ASIN `B0BQ1M98FB`, quoted); Oral-B Daily Clean Replacement Brush Heads, Pack of 3 (ASIN `B089LBBDZR`, quoted)
- Group: Assistant mistake (reviewer label `compatibility_unconfirmed`)
- Status: completed, 1 min 21.7 s
- What happened: Kept the regular electric-brush detail and chose Daily Clean 3-pack heads with the Crest in one $20.61 basket, cleanup done. Still didn't establish the handle type or caveat fit. The reply labeled a 10:15 PM local expiry as UTC (it was 05:15 UTC). One refused out-of-scope read is in the record; the quote itself worked.
- Next step: Confirm the handle type or qualify the pick; fix the time-zone wording separately.

### A6 · Amazon · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50"
- Products picked: STANLEY Quencher H2.0 FlowState Tumbler (Peony, 40 oz) (ASIN `B0CZF97C8N`, quoted)
- Group: Assistant mistake (reviewer label `soft_budget_misclassified`)
- Status: completed, 1 min 37.8 s
- What happened: Pink Peony Stanley 40 oz, full quote $88.85, above target; arrival Oct 27, before Saturday Oct 31; seller, charges, destination and payment shown; cart restore verified. The assistant sent the budget as a hard $50 cap, unlike the tester's soft-target note, stopped after one search and asked about other pinks though any pink was allowed.
- Next step: Keep target vs hard cap apart from store over-budget behavior; search within budget before asking for a preference already given. The wording is somewhat ambiguous, so one run doesn't prove a general budget bug.

### A6 · Amazon · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50"
- Products picked: STANLEY Quencher H2.0 Flow State Tumbler, 40 oz, Peony (ASIN `B0CRMP3RQT`, selected; Nook blocked on title)
- Group: Nook backend bug (reviewer label `identity_block_and_reply_gaps`)
- Status: completed, 1 min 10.9 s
- What happened: Saw a pink Peony 40 oz at $29.97 item price, but Nook blocked it because the checkout title was longer than the selected title at the same price. The released DJI fix passes; this different title case is still open. No full charges or arrival shown. The reply called it under $50 from the item price, left out Saturday, and treated the soft target as hard. Cart restore verified. Also: Reply also called it under budget from item price alone and dropped the Saturday deadline.
- Next step: Read the retained product-identity evidence for this title block before changing the identity rule again; keep the deadline and separate item price from delivered total.

### U1 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `appropriate_address_clarification`)
- Status: completed, 53.7 s
- What happened: Kept the Jumbo Jack meal, medium Coke and tip. The saved address label in the request didn't match the label on the account, and the assistant asked which saved address to use, with no made-up answer. No menu, price, tip or quote proved; no cart change.
- Next step: A durable buyer alias or evidence-based address match could avoid this question; don't guess a destination from unrelated store context.

### U1 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `unverified_home_and_store_unavailable`)
- Status: completed, 31.5 s
- What happened: Kept the meal, medium Coke and tip but left out the destination. Nook searched the account default and returned store_not_found. The reply said that was the requested saved address without evidence the labels matched; nearby results don't prove availability at the intended address. No quote, tip or cart change.
- Next step: Resolve the intended saved address before blaming store availability on it; don't switch restaurant or address without asking.

### U2 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it"
- Products picked: none; not recorded
- Group: Stopped by operator (reviewer label `unsupported_configuration_and_retry_loop_operator_stopped`)
- Status: cancelled, 4 min 43.4 s
- What happened: The first request correctly listed the two large fries as separate configurations, one no-salt. Nook refuses identical items with different options (duplicate_product). The assistant then tried four single no-salt requests; all returned item_dialog_missing, so no-salt availability stayed unknown. The operator stopped the loop after five accepted quotes; no final reply. Later reads confirmed all five blocked and settled, no cart adds, VM stopped. Also: Nook can't hold two of the same item with different options (duplicate_product).
- Next step: Separate configured line identity from product identity, and stop unchanged retries after a repeated reader failure. Not an unassisted finish.

### U2 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it"
- Products picked: none; not recorded
- Group: Nook backend bug (reviewer label `honest_unsupported_with_attribution_gap`)
- Status: completed, 43.0 s
- What happened: Kept two distinct salt configurations, hit the same duplicate_product limit and stopped after one request without merging or dropping the restriction. The reply blamed Uber Eats rather than Nook and offered two no-salt fries without evidence that option exists. No cart adds or quote. Also: Reply blamed Uber Eats instead of Nook and offered an unverified alternative.
- Next step: Explain the Nook limit accurately and label unverified alternatives as such; support configuration-aware line identity in the backend.

### U3 · Uber Eats · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one"
- Products picked: none; not recorded
- Group: Stopped by operator (reviewer label `repeated_menu_store_failures_operator_stopped`)
- Status: cancelled, 5 min 2.9 s
- What happened: The first request kept both large pizzas, but menu matching failed. Seven accepted quotes alternated product_choice_requires_user and store_not_found as the query and restaurant name changed. Recovery dropped the sizes for a while and once shrank to "pizza, quantity two", losing the toppings. The operator stopped the loop; no final reply, basket or deal. Later reads confirmed all seven blocked and settled, no cart steps, VM stopped. Also: Menu matching failed repeatedly on the Nook side.
- Next step: Show the actual observed menu and availability evidence and keep the original constraints while probing; stop repeated failures without inventing a pizza or deal.

### U3 · Uber Eats · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `menu_match_block_with_unverified_alternative`)
- Status: completed, 46.9 s
- What happened: Kept both large pizzas, then stopped after one product_choice_requires_user refusal for the pepperoni. No cheese search, deal or checkout. The reply admitted the quote was incomplete but proposed a specifically named Large 1-Topping Pizza that wasn't in what Nook returned. No cart activity. Also: Nook also stopped on a menu match for the pepperoni.
- Next step: Return observed menu candidates and keep going on independent items instead of inventing a plausible menu option.

### I1 · Instacart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you"
- Products picked: Coca-Cola Soda, 24 x 12 fl oz (Instacart product `93587`, quoted)
- Group: Quoted correctly (reviewer label `correct_quote_cheapest_delivered_unproven`)
- Status: completed, 1 min 52.1 s
- What happened: Found local offers, chose Walmart's 24 x 12 fl oz regular Coke cans ($15.52 merchandise) and kept the product and store link through the follow-up. Reconciled checkout $26.92 including service, long-distance, tax and fee lines plus tip; cart restored. Walmart had the lowest matching product price, but other stores' full fees weren't quoted, so cheapest delivered total is unproven.
- Next step: Keep product-price comparison separate from delivered-total comparison.

### I1 · Instacart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `unnecessary_retailer_clarification`)
- Status: completed, 9.1 s
- What happened: Sent no search or quote. Asked the shopper to name a retailer and said Instacart needs one, though the single-item store-discovery path was available (DeepSeek used it in the previous run). Misses the prewritten outcome of finding the cheapest store. No cart activity. DeepSeek read the Instacart-specific note; Luna read only the general files.
- Next step: Line up guide use with what the tool can actually do; ask only for missing preferences. Check what the assistant read before blaming only the model.

### I2 · Instacart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees"
- Products picked: Marketside Grade A Cage Free Large Brown Eggs, 12 ct (Instacart product `20659148`, selected; blocked before cart); Great Value 2% Reduced Fat Milk, 128 oz (Instacart product `20630136`, selected; blocked before cart)
- Group: Nook backend bug (reviewer label `conflicting_pack_evidence_unresolved`)
- Status: completed, 1 min 55.0 s
- What happened: Found both items at one Walmart and kept exact product links for the follow-up. The egg offer says 12 ct, the milk 128 oz. The follow-up called the same egg listing a pack of four and blocked before adding to cart, so the real pack size is unresolved. The assistant repeated "four" as fact and proposed quantity three. Milk found in search only, not verified at checkout; no basket or fees. Also: The assistant repeated the disputed pack count as fact.
- Next step: Trace pack-count reading against the retained source evidence before suggesting three packs; tell a candidate basket apart from a verified checkout.

### I2 · Instacart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `premature_retailer_clarification`)
- Status: completed, 10.7 s
- What happened: Asked which store to use for both items without any search or quote attempt, though same-store discovery was available (DeepSeek used it in the previous run). Nothing dropped or falsely quoted, but an extra interruption before trying the tool.
- Next step: Use store discovery before turning a tool-solvable store choice into a required question; an optional preference can be offered without blocking.

### W1 · Walmart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway"
- Products picked: Great Value Purified Drinking Water, 16.9 fl oz, 24 Count (Walmart item `13448854`, quoted)
- Group: Quoted correctly (reviewer label `correct_pickup_quote_pending_bag_fee`)
- Status: completed, 1 min 1.6 s
- What happened: One Great Value 24-count water pack, pickup as requested. Reconciled quote $5.17 ($3.97 items + $1.20 taxes and fees), cart restored. The reply disclosed the pending bag fee and listed a $5.46 authorization-hold amount separately. A pickup quote, not a booking or a placed hold.
- Next step: Keep pending fees and a possible future hold separate from the quote and any real charge.

### W1 · Walmart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway"
- Products picked: Great Value Purified Drinking Water, 16.9 fl oz, 24 Count (Walmart item `13448854`, quoted)
- Group: Quoted correctly (reviewer label `correct_pickup_quote_pending_bag_fee`)
- Status: completed, 1 min 7.5 s
- What happened: The same Great Value 24-count water pack for pickup, $5.17 reconciled estimate. The reply disclosed the pending bag fee and called it a quote; no shipping swap or booking claim. Cart restored.
- Next step: No correction needed; keep the pending-fee caveat before buying.

### W2 · Walmart · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup"
- Products picked: Great Value Everyday Strong Paper Towels, 2 Double Rolls (Walmart item `10762707702`, quoted)
- Group: Quoted correctly (reviewer label `brand_pickup_correct_pack_ambiguous`)
- Status: completed, 1 min 0.6 s
- What happened: Switched Bounty to Great Value and kept pickup. Chose one 2 Double Rolls package at $3.26 with tax, pending bag fee disclosed, cart restored. The request dropped the original "big pack" wording, so the intended size is unclear; no made-up size.
- Next step: Keep a soft pack-size wish while changing brands and show the pack chosen; don't invent a number from "big pack".

### W2 · Walmart · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `observed_pack_clarification`)
- Status: completed, 46.9 s
- What happened: Kept the switch to Great Value, pickup and the "big pack" wish. Nook found 12 triple, 6 roll and 2 roll choices but needed an exact pack; Luna asked among those real options, with no made-up size or shipping fallback. No cart activity. Meets the prewritten allowance for a pack question.
- Next step: Consider picking from observed packs for soft size wishes instead of treating a vague size as exact.

### T1 · Target · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `honest_search_block_unverified_suggestions`)
- Status: completed, 1 min 38.8 s
- What happened: Two requests kept Tide PODS 42-count and shipping; both returned product_choice_requires_user with no candidates. The assistant correctly said search failed before cart, so minimum, fees and total are unknown, with no filler or pack change. But it added unverified claims about Target's usual pack counts and unneeded internal preference details. No cart activity. Also: Nook also returned no candidate listings.
- Next step: Return observed listings or source-backed reasons for no match; avoid catalog guesses.

### T1 · Target · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `honest_product_match_block`)
- Status: completed, 59.5 s
- What happened: One request kept Tide PODS 42-count and shipping. Nook found no fitting listing and returned no alternatives. The assistant said no price or minimum was verified, didn't claim it was out of stock, and asked before changing pack or quantity. No cart activity.
- Next step: Show the candidates Nook inspected and why they didn't match.

### C1 · Chewy · DeepSeek V4.1 Flash

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that"
- Products picked: none; not recorded
- Group: Assistant mistake (reviewer label `exact_bag_unresolved_one_time_unexercised`)
- Status: completed, 2 min 49.5 s
- What happened: Two attempts kept Chicken & Rice and 30 lb; the second added Complete Essentials as a search refinement. Both blocked on product match, showing senior 7+ or other flavors. The existing cart was set aside and restored both times; no cart add or Autoship. The reply asked before substituting but wrongly said the shopper named Complete Essentials, and described the buy-once policy without reaching that checkout step. Also: Nook blocked both attempts on product match.
- Next step: Resolve the exact formula and weight from observed choices or a product link; don't credit the shopper with an inferred product line.

### C1 · Chewy · GPT-5.6 Luna

- Real prompt, scrubbed (frozen at c3d38e2; hash checked for both models): "dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that"
- Products picked: none; not recorded
- Group: Asked a fair question (reviewer label `exact_bag_unresolved_one_time_unexercised`)
- Status: completed, 1 min 9.6 s
- What happened: One request kept the exact adult Chicken & Rice 30 lb target. Nook showed senior 7+ or other flavors but no exact bag. The assistant left it unresolved and asked before changing anything, without claiming a quote or buy-once check. Existing cart set aside and restored; no cart add or Autoship.
- Next step: Coverage gap: a verified exact product is still needed to test one-time checkout.

## Task bank (example data)

| ID | Store | Shopper typed | Counts as correct | Difficulty |
|---|---|---|---|---|
| T-001 | Amazon | can you get me the dji mic mini thing from amazon | Any DJI Mic Mini kit after asking which kit | hard |
| T-014 | Walmart | need paper towels the big pack, bounty not the off brand walmart pls | Bounty Select-A-Size, 12 double rolls or larger | medium |
| T-022 | Chewy | dog food same as last time?? the purina pro plan salmon one, 30lb | Purina Pro Plan Adult Salmon & Rice, 30 lb | easy |
| T-031 | Target | get 2 of those owala water bottles 24oz, the blue-ish one my kid likes | Owala FreeSip 24 oz, any blue colorway, qty 2 | medium |
| T-040 | Instacart | bananas (not too green lol), 2 gal whole milk, eggs 18ct, whatever bread is cheapest | 4 items; cheapest sandwich loaf at that store | hard |
| T-047 | Uber Eats | chipotle bowl chicken double rice no beans, under 20 bucks w/ fees | Chicken bowl, extra rice, no beans; total ≤ $20 | hard |
| T-055 | Amazon | my airpods case died can u find just the charging case, gen 2 usb c | Apple charging case only, USB-C; ask if no case-only listing exists | hard |
| T-063 | Target | tide pods free & clear like 80ish count | Tide PODS Free & Clear, 72–91 count | medium |
| T-070 | Amazon | ink for my HP envy 6055e, black AND color | HP 67 or 67XL black + tri-color | medium |
| T-078 | Chewy | cat litter that doesnt smell, unscented clumping, 40 lb-ish, under $30 | Unscented clumping, 35–42 lb, ≤ $30 | medium |
| T-086 | Target | huggies sensitive wipes, 3 packs, whatever count is the better deal | Huggies Sensitive, lowest price per wipe, 3 packs | hard |
| T-093 | Amazon | uh the ninja air fryer thats like 100 bucks, 4qt? | Ninja 4 qt air fryer, $80–120; ask if several match | medium |