---
title: "Checkout Bench: can an AI agent buy from your store?"
description: "We send shopping agents through your store the way your customers' assistants will, and show you where they get stuck. Real signed-in checkouts, graded in code."
canonical: https://www.checkoutbench.ai/
last_updated: 2026-10-25
---

# Can an AI agent buy from your store?

> Checkout Bench: real store checkouts for AI shopping agents.

Checkout Bench runs AI shopping agents through real, signed-in store checkouts and grades every run in code. Of 527 quote requests, 186 reached a ready quote at checkout — 35.3 percent. Traces can be replayed, and every receipt shows which step passed and which one broke.

One dot per real quote request falls into a seven-column chart for where its quote paused. The columns stack to 3, 6, 0, 287, 35, 10, and 186 ready quotes (527 quote requests, Sep 26 to Oct 25, 2026).

Six store scorecards (Amazon, Uber Eats, Instacart, Walmart, Chewy, Target) show how many quote requests reached a ready quote, the median time, and the median quoted total, from the same Sep 26 to Oct 25, 2026 window.

A test roller lets you shuffle an example store, model, task and budget into a sentence someone could queue; the queue button only shows a confirmation and connects to nothing.

A phone replay walks an example store from sign-in to a ready quote of $14.13 under a $30 cap, using the real median step times (8.2, 16.2, 23.3, 39.7 and 43.9 seconds). The store screens are example data.

A run graph shows example case KW-color-01 on Amazon as five zones, from the entrance to the checkout counter. The entrance records a color choice the agent handed back to the person.

Example runs in progress: a run card walks through its steps and lands on a ready quote. The counter and the queued runs beside it are example data.

Four lab models beam a shopping task to the Nook agent, which fans out to Amazon, Uber Eats, Instacart, Walmart, Chewy and Target. The store in focus shows its real count of ready quotes: 101, 34, 23, 13, 10 and 5.

A graded receipt for the Crest Pro-Health Whitening Gel Toothpaste run on Amazon of Oct 25, 2026 (DeepSeek V4.1 Flash, quote only) shows each field with the check it feeds. The total $4.33 is $3.93 + $0.00 + $0.40.

The full set of results charts from the test suite: lab runs per day, run outcomes, quotes by store, why quotes were blocked, time to a quote, where the time goes, trace events, assistant models, quoted totals, listing price versus quoted total, store coverage, test suites passing, and changelog activity. Every chart is real.

A split view shows 527 quote requests stepping down to 186 ready quotes (35.3%), with the bars growing in on view. The largest drop is from search results to checkout read (518 to 231).

An example model leaderboard ranks GPT-5.6 Luna, DeepSeek V4.1 Flash, Grok 4.7 and Gemini 3.8 Flash by made-up ready-quote rates with 95% intervals, using the names from the site's own example bump chart. It is not a published ranking.

A large 186 counts up on view: ready quotes reached checkout, 35.3% of 527 requests.

A paper-style draft sits beside a KPI strip of real headline counts (596 assistant runs, 527 quote requests, 186 ready quotes) and an example model leaderboard with made-up 95% intervals for GPT-5.6 Luna, DeepSeek V4.1 Flash, Grok 4.7 and Gemini 3.8 Flash.

A failure taxonomy shows the top ten blocker codes among 336 blocked quotes, grouped by whose problem it was. The other 89 blocked quotes are not broken out.

A store leaderboard ranks those six stores by the share of requests that reached a ready quote (real data). A try-your-store demo walks the checkout stages and lands on an example readiness score; it is not connected to a live test.

A person-and-agent split shows the same example order run two ways, both reaching a ready-to-pay review at $351.20. The store, timings, and total are example data, not a live order.

## Key findings

- **35%** of 527 real quote requests ended in a ready quote. The biggest drop is from search results to checkout read: 518 to 231. (Figure 1)
- **11 of 30** reviewed runs went wrong on the assistant's side, 5 on Nook's, and 7 quoted correctly. (Figure 2)
- **40** stores and sites can be connected today, and 7 return a full cart and total. (Section 04)

## Can an AI agent buy from your store? (for retailers)

**We send shopping agents to your site and show you where they get stuck.**

People are starting to hand their shopping to AI assistants. When an assistant can't find the right size, gets lost in a pop-up or stalls at checkout, the sale just doesn't happen, and nobody tells you. We find those places before your customers do.

### Where agents get stuck

A flow of who runs and where they go shows four lab models handing a task to the Nook agent, which fans out to the six stores and returns each store's real ready-quote count, median quoted total and median time.

- **Search.** The agent searches the way a person talks and gets nothing useful back. What we check: Do ordinary requests land on the right product?
- **Product options.** Size, color or pack count is chosen in a way agents can't read, so they stop or pick the wrong one. What we check: Can an agent pick the variant the person asked for, even when the store calls “green” “Lime”?
- **Sign-in.** Logins, pop-ups and account checks block the agent before it reaches the cart. What we check: Does a signed-in session get through without a person stepping in?
- **Cart.** Quantities, bundles and add-ons end up different from what was asked. What we check: Does the cart hold exactly the right item and amount?
- **Checkout.** Fees, delivery choices or a final review screen the agent can't get past. What we check: Does it reach the last screen before paying, with the right total?

### What you get back

- A score for each step, from search to checkout
- A replay of every place an agent got stuck (coming soon)
- A fix list, ordered by how many runs each problem blocked
- Every result split into “your site” and “the agent”, so you only fix what's yours
- Re-checks: After you ship fixes we run the same requests again, and on a schedule after that. If a site change starts breaking agents, you hear about it with the step that broke.

### A private test environment built on your store

1. We work with your team and your own product data: your catalog, your variants and your real checkout flow.
2. We build a test setup that is yours alone, with the requests your customers actually make.
3. We run the assistants your customers are starting to use, such as ChatGPT, Gemini and Perplexity, through it and show exactly where each one gets stuck.
4. Nothing is ever bought, and your data stays in your environment.

**Why trust the numbers.** The same runs power our public benchmark. Every run happens in a fresh cloud browser on a real store, stops before paying, and is graded in code, not by another AI.

**For developers.** The engine that drives these runs is also available as an API for teams building assistants that shop. Details are further down the page.

## What exists today (real)

- **40**: stores and sites a person can connect an account to today: 12 built in, 28 set up from their web address
- **7**: stores where a full cart and total come back today: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE
- **1**: fresh cloud browser per run, thrown away afterwards
- **384**: trace events in one real run: DeepSeek V4.1 Flash asked Amazon for a DJI mic; Nook blocked all five quotes as product mismatch, nothing bought

## Charts on this page

- **Quote funnel (real):** Quote requested 527 (100%) → Browser ready 524 (99%) → Signed in 518 (98%) → Search results 518 (98%) → Checkout read 231 (44%) → Quote checked 196 (37%) → Ready quote 186 (35%). Source: Nook quote lab run records (458 runs, Sep 26 to Oct 25, 2026 PDT), aggregated read-only.
- **Whose problem it was, by store (real, Test flight, 30 runs):** Amazon (12): Quoted correctly 3, Asked a fair question 1, Assistant mistake 5, Nook backend bug 3, Stopped by operator 0; Walmart (4): Quoted correctly 3, Asked a fair question 1, Assistant mistake 0, Nook backend bug 0, Stopped by operator 0; Target (2): Quoted correctly 0, Asked a fair question 1, Assistant mistake 1, Nook backend bug 0, Stopped by operator 0; Instacart (4): Quoted correctly 1, Asked a fair question 0, Assistant mistake 2, Nook backend bug 1, Stopped by operator 0; Uber Eats (6): Quoted correctly 0, Asked a fair question 1, Assistant mistake 2, Nook backend bug 1, Stopped by operator 2; Chewy (2): Quoted correctly 0, Asked a fair question 1, Assistant mistake 1, Nook backend bug 0, Stopped by operator 0.
- **Lab runs per day (real):** Sep 26 76, Sep 27 39, Sep 28 12, Sep 29 25, Oct 1 264, Oct 2 13, Oct 25 29; no runs on the other days from Sep 26 to Oct 25, 2026 PDT.
- **Model rank by store (example data):** GPT-5.6 Luna 2/1/2/3/1/2; DeepSeek V4.1 Flash 1/2/3/1/2/1; Grok 4.7 3/4/1/2/4/3; Gemini 3.8 Flash 4/3/4/4/3/4 (ranks at Amazon, Walmart, Target, Instacart, Uber Eats, Chewy).
- **Price check ladder (example counts on the real KW-color-01 case):** reference $349.00, grader allowance ±$6.98 = max(2% of the pre-tax item subtotal, 50¢).
- **Quoted totals through a day of checks (example counts on the real KW-cable-01 case):** totals near $28.12 until a made-up $2.99 shipping fee at 15:00 pushes every quote over the $30 cap.

## Five directions

### Option A (Benchmark): How well do AI agents actually shop?

We hand models the requests real people type, and let them work inside real, signed-in store accounts at Amazon, Walmart, Target and four more. Code checks the final cart. We publish what passed and what didn't.

- **Who buys it:** Model labs and agent companies that want an outside number for shopping tasks, and the people choosing between models.
- **What they get:** A public leaderboard by model and by store; Private runs on models that aren't released yet; The full step-by-step trace behind every pass and fail
- **How it works:** A request written the way people write: “can you get me the dji mic mini thing from amazon” → The agent works in a fresh cloud browser on the real store and stops before paying → Code checks the item, the quantity, the price cap, and that nothing was bought
- **Pricing idea:** Leaderboard is free to read. Private evaluation rounds are priced per model, per round (not set yet).

### Option B (RL training environment): A practice store for training shopping agents

Recorded copies of real store pages that reset instantly, a bank of messy requests with the right answers written down, and graders written in code. For labs training agents that buy things.

- **Who buys it:** Research and post-training teams at model labs.
- **What they get:** A container with a reset / step interface that runs many copies in parallel; Labeled tasks: each one lists the products that count as correct, or says the right move is to ask; Graders in code, no model judging another model; A held-out set of live runs to check the copy still matches the real store
- **How it works:** Record real sessions at each store, with personal details removed → Replay them as a store that resets in milliseconds and behaves like the original → Score every episode in code and keep the live exam separate from training
- **Pricing idea:** Licensed as task packs. Comparable deals we have seen land around $200–2,000 per task depending on difficulty and exclusivity. A range, not a quote.

### Option C (For retailers): Can an AI agent buy from your store?

We send shopping agents to your site with ordinary requests and show you where they get stuck: search, product options, sign-in, cart or checkout.

- **Who buys it:** E-commerce and growth teams at retailers and brands, and the platforms that host their stores.
- **What they get:** A score for each step from search to checkout; A recording of every place an agent failed; A fix list ordered by how many runs each problem blocked
- **How it works:** Pick 20–50 requests your customers really make → Several different agents try each one in fresh browsers → Read the report, fix things, run it again
- **Pricing idea:** A flat fee per audit, with an optional monthly re-check.

### Option D (Monitoring): Know when a store change breaks shopping agents

We run a fixed set of shopping tasks against each store on a schedule. When a site change makes them fail, you get an alert with the step that broke and a recording of it.

- **Who buys it:** Teams that ship shopping agents or browser automation, and retailers who want agents to keep working on their site.
- **What they get:** Scheduled runs per store; Alerts by email, Slack or webhook; A before-and-after comparison of the step that broke
- **How it works:** Choose the stores and tasks you care about → We run them every few hours in fresh browsers → When something that passed starts failing, you get the step, the screenshot and the difference
- **Pricing idea:** Monthly, per store watched.

### Option E (API): One API to sign in, fill a cart and check out

Your agent sends what the person asked for and which connected store to use. We sign in, find the product, build the cart and send back the total. Nothing is bought until the person says yes.

- **Who buys it:** Developers building assistants that need to buy things for their users.
- **What they get:** Account connection at 40 stores and sites; Cart and total at 7: Amazon, Uber Eats, Instacart, Walmart, Target, Chewy, DETOURE; Orders placed only after the person confirms; A full trace of every request
- **How it works:** The person connects a store account once → Your agent calls the API with their request → You get the cart and total back, and place the order only after they confirm
- **Pricing idea:** Paid per cart total returned and per order placed (rates not set yet).

## FAQ

### Is this a real product?

Yes. Checkout Bench runs AI shopping agents through real, signed-in store checkouts and shows where they get stuck. Retailers can request a private test of their store, the test suite at /lab lets signed-in users run assistants against real quotes, and agent developers can use Nook's API, documented under Nook docs.

### What exists today?

Account connection at 40 stores and sites (12 built in, 28 set up from their web address); a cart and total at 7 (Amazon, Uber Eats, Instacart, Walmart, Target, Chewy and DETOURE), and at the first six the order itself once the person says yes; a fresh cloud browser for every run; a full trace of each run; and the test suite at /lab. One real run, DeepSeek V4.1 Flash asked for a DJI mic on Amazon, recorded 384 trace events, and all five quotes were blocked as product mismatch (the checkout title was longer than the product page's).

### How is a run graded?

In code, from the run's record: right item, right quantity, within any budget the person gave, and a question when the request was really unclear. A quote test passes only if it stops at the quote; a purchase counts only once the store confirms the order. No model grades another model.

### Does anything get bought?

Only after a person says yes. Ordinary tests stop at the cart total and put the cart back. A purchase needs the person's explicit yes to that quote's total, and it counts only once the store confirms the order.

### Are the numbers on these pages real?

The facts, charts and recorded runs are real and marked Real. The leaderboard, the task bank and the small previews on the five cards use example data and are labeled Example data.

See also: [test suite](/suite.md), [llms.txt](/llms.txt)