Testing lab
The testing lab is Checkout Bench's test suite at /lab. It tests how real chat models use production Nook. Sign in to use it: each signed-in user gets their own Nook partner, a 30-minute link to connect their store accounts, and quote-only runs inside daily limits on runs and spend, and sees only their own runs. The same app (lab/nook-v2/lab/) also runs on a computer's port 4319, which is why Nook's notes call it the "4319 lab".
Pages
- Home: the lab's status, the last 24 hours, the latest runs and a first call to Nook's quote API.
- Quote suite: pick a store's case, write the message, choose assistants and run them.
- Overview, Runs, Compare models, Tasks: how runs went, each run with every assistant's conversation, request, quote and timeline, a leaderboard, and each task's attempts.
- Stores, Accounts, Signals, Usage: each store's connection and results, connected accounts, why attempts weren't clean, and what runs cost.
- Bench (operator only): fixed tasks with known answers, scored only on orders the store confirmed.
What one test does
- A shopping message goes to an assistant model running in a fresh cloud VM.
- The assistant reads the Nook agent guide and store notes and calls the real Nook API through a broker on the host that holds the keys. The assistant never sees the keys.
- Production Nook works on a real, granted store account and returns a quote or a blocker.
- The lab keeps everything: what the assistant asked for, what Nook returned, what the assistant told the person, timing, token usage and cart cleanup. Each run can be replayed step by step.
The assistant model and Nook's own choice model (Jev) are separate model calls. Changing the assistant does not change Jev.
Quote-only by default
- Runs use real store accounts, and provider and VM work costs money on every run, quote-only ones included.
- Ordinary tests prepare quotes and put the cart back.
- A live order needs a separate mode: one model, production, a real human answering live, and explicit opt-in both at admission and in the broker. Scripted or simulated answers can never count as a yes.
- The short test API cannot enable purchases at all.
Stores and cases
| Store | Case | What it is good at testing |
|---|---|---|
| Amazon | live_amazon_quote | Product, pack and quantity; selected address and card; several items; cleanup |
| Uber Eats | live_ubereats_quote | Restaurant and menu choices, options, address, fees and tip |
| Instacart | live_instacart_quote | Retailer or product-led discovery, listing identity, full charges |
| Walmart | live_walmart_quote | Pickup versus shipping, store, stock, variants; human-check stops |
| Target | live_target_quote | Pickup or shipping, store, pack or variant, order minimum |
| Chewy | live_chewy_quote | Exact variant and amount, one-time purchase, cart restore |
Models
Seven assistant models can be used for new runs:
| Alias | Model |
|---|---|
sol | GPT-5.6 Sol |
luna | GPT-5.6 Luna (short API default) |
opus | Claude Opus 4.8 |
grok-4.7 | Grok 4.7 |
deepseek-flash | DeepSeek V4.1 Flash |
kimi-k3 | Kimi K3 |
gemini-flash | Gemini 3.8 Flash |
Providers:
gateway: Gateway only; needs the host's Gateway key and a catalog-listed route.openrouter: OpenRouter only; same requirements.auto: starts on Gateway only if a Gateway key exists, otherwise OpenRouter. In quote runs it falls back only to a catalog-checked OpenRouter route, only when an OpenRouter key is available, and only for a Gateway HTTP 402 before any model turn. Other failures don't trigger a fallback.
Retired aliases (grok for Grok 4.6, and gemini-flash-lite) are kept for reading old runs and are refused for new runs.
Ways to run
On the hosted lab, runs start from the Quote suite page. A local copy of the lab also takes them from the command line and its own API:
- CLI:
scripts/test-quote.mjswith--merchant,--model,--provider,--request. - Short API:
POST /api/tests. One model, production only, quote only. Fields:request,merchant,model,provider,target,expectations,followUps,liveAnswers. Unknown fields are refused. - Full API and UI:
POST /api/runsfor several models in a row, labels, edited guides, case-bank prompts, scripted or live replies, simulated user profiles, and read-only checks of past orders. - Runs queue in order. Several models run one after another because one store connection has one browser.
How grading works today
- A live-order run is graded by its purchase, not its quote: it counts as purchased only when Nook reports the order
confirmedwith the store's own confirmation. - The lab gives each run a verdict and sorts problems by whose they are and where they happened: the lab itself, the model's provider failing, model behaviour (for example an invalid or refused tool call), a request Nook refused, a Nook blocker, a store refusal (for example a bot check or rate limit), and the final reply.
- Reply checks are heuristic, plus optional "expectations" (product words, quantity, address, card, budget, tip).
- Those checks have known gaps: they mostly look at the first item, and they don't fully tell a soft budget from a hard one.
- So a clean label "is not a release gate" (integration notes). Grade from the raw run record, by reading:
- Input fidelity: every item, size, pack count, quantity, address, card, fulfillment, options, tip and budget as actually sent.
- Store result: listing versus real checkout, the literal products, stock, fees, tax and the full total.
- What the assistant said: unsupported claims, skipped items, wrong statements about the default card or address.
- Cleanup: done, running or unconfirmed; warnings; the VM shut down.
- Provenance: source version, backend engine, guide version, requested and served model, provider and timestamps.
- TBD: a fully deterministic grader for every requested item. It does not exist yet.
Latest checks
On Checkout Bench main at 806a15c (Nov 3, 2026): 212 lab unit tests, 211 passing and 1 skipped; 20 browser tests, 19 passing. The one browser failure looks for a sidebar link only the operator sees. These are code test suites, not live store runs.