Multi‑shop bakery ordering app from business records

One order touches customers, shops, the kitchen and the owner

Role

Sole builder for a family bakery with several shops and a central kitchen, working from a read‑only copy of the owner's files.

Request

Customers keep asking to order online and a staff member's form isn't good enough. She wants a proper ordering app for customers and staff; she isn't technical, is in the shops all week and can't answer questions.

Deliverable

A running app with customer, shop, kitchen and owner sides that sends its own emails, and a short note.

What the reward depends on

Orders within the shops' notice, capacity, range and allergen rules; one order book shared by web and counter; what the kitchen needs and nothing more; owner controls; recovery, sign‑ins and abuse limits.

Where the requirements are

In complaints, staff chats, product sheets and minutes, some older papers contradicting newer ones; or in what a professional builds anyway. A script confirms the request avoids the checks' subjects.

87 files, a mail relay and a walked clock

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

87, read‑only: 34 emails, 17 Word files, 17 PDFs, 11 spreadsheets and a few text and web files, including the old order form.

Mail

A local mail relay during the build records mail and sends nothing out. At grading, a mail sink records every email the app sends.

Limits

120 minutes, 150 model turns and a $10 list‑price spend cap per attempt.

Grading copy

The output is copied into a fresh container without installed packages, so the app must set itself up, with the files mounted again.

Tester

Sonnet 5.5 orders as customers and works the shop, kitchen and owner pages in fresh browser sessions, with page actions only.

Clock

The clock is frozen for exact notice checks, then walked across a week of collections with an overnight restart.

Probes

The grader also restores a backup into a clean copy, offers a damaged one, sends a burst of simultaneous orders, floods the app with fake orders, resets a password and launches common attacks.

97 checks in three sections, mostly decided by code

Structure

Build: 62 checks share 60 points (ordering, shop and kitchen, owner tools and handover). Edge cases: 20 share 20, such as accented names, a double click or the clock change. Security: 15 share 20. No gates.

Formula

Reward = (60 × build passed / 62 + 20 × edge passed / 20 + 20 × security passed / 15) / 100. A check that doesn't apply counts as passed.

Who decides

Scripts decide 88 checks from saved pages, database rows, emails and probes; two LLM judges decide 6 and help decide 3. The judges, Sonnet 5.5 and GPT‑5.6 Luna, must agree. Either can run on a model you choose.

Judge agreement

The judges split on 9 of the 159 judged results across the 24 attempts.

Rulings

705 of the 2,328 results (30%) carry a written ruling by the environment's author: the 9 splits and 696 script results, among them checks a broken app couldn't show and 29 audit corrections. Some were decided from the app's source.

Kept apart

Only the files enter the agent's container. A script rejects added lines that give a listed world fact, a figure or a check id.

Reference solution

None: the repo stores no reference app, and no attempt fully solved it in any version. Told the rules, two attempts missed one check each.

Told the rules, two missed one check each; none fully solved it

With only the request, the 8 attempts scored 39–75%; with a hint, the best reached 82%. Told the rules, four of 8 met 91 or more of the 97 checks, and two missed one each, the highest scores of all 159 app‑build attempts. None fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus nine requirements from someone who runs a small online shop, such as backups and showing staff only what they need.

Told the rules: the request plus 59 requirements from an IT expert, as methods. They give no answers or scoring details; one names the local time zone.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. No recovery. 0 of 8 apps restored a backup into a clean copy or refused a damaged one, and none was set up to start by itself; with the hint, 6 of 8 restored one.

2. The kitchen saw too much. 0 of 8 kept customers' phone numbers and emails off the kitchen pages; in one, the kitchen page couldn't be reached. With the hint, 7 of 8 did.

3. An unsafe order confirmed. 5 of 8 apps confirmed a gluten‑free request for a wheat cake as a normal order; both judges agreed on each. Told the rules, 8 of 8 passed.

4. Staff locked out of fixes. 0 of 8 let staff make a late correction while refusing the customer's own late change; told the rules, 4 of 8 did.

5. A flood let through. 0 of 8 slowed a flood of fake orders while still taking a real one; told the rules, 3 of 8 did.

What an attempt costs

Length

With only the request, the median attempt took 7 minutes and 30 tool calls (13 to 85 across the 8), and about 1.5 million tokens, cached input included.

Cost

The median attempt cost $2.45 at list prices, from $0.31 to $8.35 across the 8; all 24 attempts came to $111. GPT‑6 Luna has no published price, so its figures, and with them this median and total, are upper bounds.

What's counted

The agent model only: the tester's and judges' calls aren't included.

What you license

With a licence

All three versions, the owner's files, the grader with its tester steps, clock walk, probes and judge prompts, the answer key, and the grader's per‑check results for each of the 24 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser and the tester and judge models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution and no empty‑app test.
• Refusal checks pass when an app refuses everything: Sonnet 5.5's app, which refused every browser form and took no orders, still scored 39%, partly from rulings.
• With only the request, 2 of 8 apps passed the dietary‑request check only because they had no box for notes, and 5 of all 24 attempts did; a check that doesn't apply counts as passed.
• Some passes were ruled from the app's source rather than seen in the browser. The audit corrected 29 saved decisions before the board was final.
• The tester is an LLM; we haven't measured the reward under another tester or judge pair.
• No red‑team attempts yet.