Laundry‑shop operations app for owner, hosts, workers

Run a shop whose rules sit in its books

Role

Sole builder for a small laundry serving short‑let hosts and walk‑in customers, with hourly collection rounds, working from a read‑only copy of the owner's files.

Request

One app for host bookings with collection slots, wash tracking and ready notices, worker check‑ins with photos, customers, invoices, payments, tax, supplies and bills. Owner, hosts and workers use it at once, so changes must show for everyone.

Deliverable

A running app with owner, host and worker views, wired to the shop's test services, and a short note.

What the reward depends on

Records imported as they are; owner controls to keep every kind of record current; each customer's terms and each building's rules; payments counted only when confirmed; safe sign‑ins; recovery and safe concurrent edits.

Where the requirements are

In 53 plain‑text books, agreements, price lists, notices and staff cards: two customers share a name, some orders are half done, a washer is out of service.

53 files, three test services and a walked clock

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

53 plain‑text files, read‑only: 31 text files, 15 emails, 6 CSV books and one Markdown file, in 11 folders.

Services

A local test service for invoicing, card and mobile checkout with signed confirmations, and SMS and email notices. It can't move real money or send real messages.

Limits

120 minutes, 150 model responses and a $20 list‑price spend cap per attempt.

Grading copy

The whole output, dependencies included, is copied into a fresh container with her files mounted again.

Tester

GPT‑6.1 Sol follows the note and signs in as owner, counter worker, driver and hosts: books, collects, washes, hands over, uploads photos, issues invoices and pays. It records values from the page and never passes verdicts.

Stages

16 in order, from start and import through customers, rounds, washing, handover, billing, stock and exceptions to edge cases, security, clock, restart, offline, recovery and a moved data folder.

Faults

The grader walks the clock past a daylight‑saving change, cuts the internet, makes the services fail or lose replies, replays a payment, damages a backup and interrupts a restore.

161 checks, mostly decided by code; Opus 5.5 rules 73% of results

Structure

Build: 114 checks share 60 points. Edge cases: 23 share 20, such as a double click, a service going down or the clock change. Security: 24 share 20. No gates.

Formula

Reward = (60 × build passed / 114 + 20 × edge passed / 23 + 20 × security passed / 24) / 100. An app nobody can use, or no note, fails each check.

Who decides

Code decides 154 checks from fixed trials and the services' records. Two LLM judges, GPT‑6.1 Sol and GPT‑6 Luna, read the evidence for the other 7 and for any check code leaves open. Either can run on a model you choose.

Rulings

Opus 5.5 rules splits, judge faults and security checks, from the evidence or the app's source: 2,455 of the 3,381 results (73%) carry its ruling, 767 of them on the five unusable apps.

No free passes

A feature with no control in the app fails each check that needs it. Missing evidence stays open; it never becomes an automatic pass or fail.

Kept apart

No key, grader, reference app or design file enters the agent's container.

Reference solution

None stored or graded, and no attempt fully solved it in any version. The best scored 81%, told the rules.

Five helped attempts left nothing usable; none fully solved it

With only the request, the 7 attempts scored 16–77%. With a hint the median fell to 10%; told the rules, four of 7 scored 74–81%. Five helped attempts scored 0: limits hit before the note, an app that never loaded, one that refused every form.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version. 16 of 21 attempts earned some reward.

With a hint: the request plus six points from a friend who knows the work.

Told the rules: the request plus 70 requirements from an IT consultant. They give no business facts, answers or scoring details.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request, unless a line says otherwise.

1. Records she can't keep current. 0 of 7 apps let the owner add, edit and archive flats, buildings, prices, machines, suppliers, stock or bills. Told the rules, GPT‑6 Astra's and Opus 5.5's passed all nine record controls.

2. No recovery. 0 of 7 apps restored a backup, refused a damaged one, or survived an interrupted restore; told the rules, 4 of 7 did.

3. Readable passwords. 5 of 7 apps left passwords readable or had no sign‑in. Told the rules, which asked outright, none of the 7 passed.

4. Money that doesn't add up. 0 of 7 applied payments to the right invoice with exact balances, or exported a month the accountant can reconcile; told the rules, 3 of 7 got the balances right.

5. Forms that saved nothing. Most of Fable 5.1's forms reloaded the page instead of saving, losing it 60 of 161 checks; it reported that every screen loaded without errors.

What an attempt costs

Length

With only the request, the median attempt took 39 minutes and 43 tool calls (14 to 171 across the 7), and about 4.7 million tokens, cached input included.

Cost

The median attempt cost $6.15 at list prices, from $1.05 to $16.57 across the 7. Cost is recorded for 20 of the 21 attempts, which came to $166. GPT‑6 Luna's figures are upper bounds.

What's counted

The agent model only: the tester's, judges' and rulings' calls aren't included.

What you license

With a licence

All three versions, the 53 files, the test services and their fault controls, the grader with its tester steps, probes and judge prompts, the answer key, and the grader's per‑check results with reasons for each of the 21 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser, the tester and judge models and the ruling model.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution stored or graded, and no empty‑app test. In our attempts the five unusable apps scored 0, by the rule that an app nobody can use fails each check.
• 73% of results carry an LLM's ruling, 66% in the 16 usable attempts. GPT‑6.1 Sol was tester and judge and built apps here; Opus 5.5 ruled and built apps. We haven't measured whether that favours them.
• Two attempts with only the request were first graded by an earlier grader, then reviewed by hand to the board standard. Separately, a review moved 13 notice decisions in seven attempts from fail to pass.
• In one attempt the grading browser stopped, and 54 checks were ruled from the app's code; some payment checks were too, after a service lock.
• Two checks were passed by no attempt in any version: damage claims and access‑card custody.
• No red‑team attempts yet.