Gym operations app with three owner‑feedback repair rounds

One app for everyone at the gym, repaired after each of the owner's visits

Role

Sole builder for an independent gym, working from a read‑only copy of the owner's 84 files, with a local payment test service on its network.

Request

Members, coaching and meal‑photo review, bookings, prepaid session‑pack balances and trainer pay, payments and accounts, equipment‑servicing reminders, QR‑code door entry with a live crowd count, community and chat, backup, a corporate client's staff contract, and every old record brought in.

Deliverable

The running app as left after the last finished round, the build or a repair, with views for staff, members and the client's HR, and a short note the owner and the grader use to sign in.

What the reward depends on

Rules and figures found only in the files, such as trainer deals and the bank contract; old records rebuilt and merged right; professional defaults such as moving a booking or refusing a damaged backup; security.

Where the requirements are

In 84 files: notebooks, workbooks, chats, bills, contracts and a photo of the paper service log. 86 checks rest on facts found only there.

84 files, a payment test service and an LLM‑played owner

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI). Internet is on for packages; web search and fetch are off.

Files

84, read‑only: 64 text files, 9 CSV, 4 PDF, 2 spreadsheets, 4 images and the payment service's guide in Markdown. One photo of the paper service log decides two checks.

Payments

A local test payment service on the app's network, in the local currency, during the build and grading.

Simulated owner

Sonnet 5.5 plays the owner: three browser visits of up to 10 minutes, after the build and the first two repairs, each ending with a short message of at most three concerns, which is all the agent receives.

Grading

An operator model, Opus 5.5, works the final round in 12 fresh sessions, one per grading stage, from the note, moving the clock and replaying payment results where checks ask.

Limits

The first build: 120 minutes and 150 model responses. One $20 list‑price cap covers the build and the three repair rounds.

113 checks of a point each, on the last finished round

Structure

113 checks: members, coaching and community (27), bookings, the door and upkeep (16), money and the bank contract (22), edge cases (24) and security (24).

Who decides

Scripts settle exact values; two judges, Sonnet 5.5 and GPT‑5.6 Luna, decide the rest, and the grading agent settles their disagreements. Of 1,921 final marks, 195 came from scripts, 1,617 from judges and 17 from tie‑break rulings.

An exception

92 marks were collected and graded by the grading agent itself, a Codex session, under an exception the benchmark's maintainer allowed; they aren't independent judge agreements.

Unseen behaviour

Grading makes new test files, such as three bad backups, two of them cut from the app's own backup, and a GPS‑tagged photo, and moves the clock, so features that are only described don't pass.

Formula

Reward = checks passed / 113, on the last finished round. An unusable app fails all 113 checks, security included; an attempt that finishes no round counts as 0.

Five apps scored 71% to 86%; none fully solved it in any version

With only the request, five apps scored 71% to 86%, one 12%, and one attempt handed in nothing. With a hint the mean fell from 60% to 55%, and told the rules it was 59%.

Each dot is one attempt; the line marks the median.

One scored attempt per model and version; 19 of the 21 were made again after faults on the evaluator's side, never after a model's own failure. 4 of the 21 finished no round and count as 0.

With a hint: the request plus 7 points from a friend who knows the work.

Told the rules: the request plus 33 requirements from an IT consultant, never facts from the owner's files.

What the models got wrong

Counted over the 6 apps graded with only the request; the seventh attempt handed in nothing.

1. No way to move a booking. 6 of 6 apps couldn't move an existing booking under the same clash and opening‑hours rules.

2. Damaged backups taken in. 5 of 6 didn't refuse a damaged or incompatible backup while keeping the data, though only 1 of 6 failed to restore a good one.

3. Old records lost or mixed. 5 of 6 apps didn't show every imported member with their plan, dates, freezes and coach, and 4 of 6 mishandled repeat rows or real identity conflicts.

4. The bank's invoice. 5 of 6 got the contract's deductions for equipment out of order and overdue servicing wrong: one merged them into a single line, one charged units not yet due, and three produced no deduction lines.

5. The door. 5 of 6 didn't refuse a second entry from a member already inside.

6. Secrets and erasure. 5 of 6 failed the check on keeping secrets out of pages, logs and exports, though only one was shown exposing them; 5 of 6 failed erasure on request that keeps the money records, four with none at all.

What an attempt costs

Length

With only the request, the median attempt took 35 minutes and 80 tool calls (60 to 164 across the 7).

Cost

The median attempt cost $8.83 at list prices, from $0.10 to $17.86 across the 7; all 21 attempts came to $176. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the owner's visits and the operator's and judges' calls aren't included.

What you license

With a licence

Revision 1 in all three versions: the owner's 84 files, the payment test service, the owner's and operator's prompts, the grader and its judges, and the grader's report on each of the 17 graded attempts, with the records of the 4 that finished no round. No reference app is stored.

Format

Runs in Docker, the app and the payment service on one network, graded through headed Chromium. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and judges' instructions aren't published.

What we haven't measured, and what's uneven

• One scored attempt per model and version, so we haven't measured how much attempts vary here.
• No reference app is stored. The self‑test uses a fake operator and fake judges, and requires an unusable app to fail all its checks.
• Most marks come from model judges: 1,617 of the 1,921, with 17 tie‑break rulings.
• The owner's visits are model‑played, so the repair rounds depend on what the simulated owner happens to notice.