Music‑teaching studio app: lessons, billing, media

She names the features; her files hold the rules behind them

Role

Sole builder for a teacher's home studio, serving her, her pupils and their parents, working from a read‑only copy of her files.

Request

Attendance, plans, notes, uploads and in‑lesson recording on each pupil's page; logins for pupils; notes for parents; invoices and online payment; records for her accountant; books sold at shop price with her discount kept private; if possible, recordings turned into sheet music.

Deliverable

A running app with teacher, pupil and parent sides, payments and media, and a short note.

What the reward depends on

Current pupils, dated prices and terms, and consent read right; every invoice exact; payments matched even under another name; privacy between families; recording, sheet marks, backups, a password reset and later imports.

Where the requirements are

In 416 files that don't all agree: an old pupil list, terms that changed partway through the year, payers under other names, families' different consents. Basics such as backups and delete confirmation aren't in her request.

416 files, a test payment service and five months

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

416, read‑only: 154 PDFs, 150 emails, 85 text files, 19 CSV exports, Word files, a spreadsheet and five audio and video recordings that make sense only when played.

Payment service

A local test payment service with hosted checkout and signed confirmations; it can't move real money.

Limits

120 minutes, 150 model responses and a $20 list‑price spend cap per attempt.

Clock

The request is dated early November, but the build container showed the real date, 19 or 20 October. At grading the clock starts at the request's date and moves about five months on.

Grading copy

The whole output, dependencies included, is copied into a fresh container with her files mounted again.

Tester

Sonnet 5.5 or GPT‑6.1 Sol uses the app as teacher, pupils, parents and a cover teacher, with a fake microphone and camera: records, uploads, pays in the test checkout, adds new bank and parent records.

Stages

14 in order: start, load, studio, billing, learning, media, sheet marks, edge cases, security, clock, restart, recovery, 20 requests at once, and a password reset.

180 checks, most of them read by two LLM judges

Structure

Build: 127 checks share 60 points. Edge cases: 23 share 20. Security: 30 share 20. No gates.

Formula

Reward = (60 × build passed / build total + 20 × edge passed / edge total + 20 × security passed / security total) / 100, over the checks that apply.

Absent features

An edge or security check on a feature the app doesn't offer leaves that attempt's totals, so attempts were scored on 155 to 179 checks with only the request.

Who decides

Scripted probes decide 11 security checks and a load test. Two LLM judges read the evidence for the other 168, and for any probe left open: GPT‑5.6 Luna and either Sonnet 5.5 or GPT‑6.1 Sol. Either can run on a model you choose.

Judge agreement

Where both verdicts are kept, the judges disagreed on 143 of 1,630 judged results. In the 19 graded attempts, 949 of 3,306 decisions carried a split flag; at the audit, most such flags were one judge finding no evidence.

Rulings

1,709 of the 3,666 results (47%) carry a written ruling, from Claude sessions and the author under the author's rescoring policy: 143 splits, 764 where one judge gave no verdict, and others the policy overrode.

Kept apart

No key, grader, reference app or private media enters the agent's container. A script rejects added lines that give a figure, certain names from her files or a check id.

Reference solution

None stored or graded, and no attempt fully solved it in any version. Solvability rests on the best attempts: 94% told the rules.

Five attempts scored 75–81%; told the rules, the best three 93–94%

With only the request, five of 7 attempts scored 75–81% and two under 36%. With a hint, the best reached 87%. Told the rules, the best three scored 93–94%. Sonnet 5 scored 0 in both helped versions, with no start script. None of the 21 fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version. 19 of 21 attempts earned some reward.

With a hint: the request plus six points from a friend on what matters most.

Told the rules: the request plus 33 requirements from an IT consultant, a revision with four lines tightened before launch. They give no business facts, answers or scoring details.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request, unless a line says otherwise.

1. No way back. Only 1 of 7 apps restored a backup and refused a damaged one, and no note said where data and backups were kept; with the hint, 5 of 7 notes did.

2. Locked out. Only 1 of 7 apps let the teacher reset her own password; told the rules, 6 of 7 did.

3. Frozen on handover day. Only 1 of 7 took in new bank files and parents' messages through its pages; told the rules, 4 of 7 did.

4. Consent not followed. Of 6 apps scored on it, only GPT‑6 Astra's stopped new video capture once a family withdrew consent. Told the rules, none of 6 did.

5. Leavers kept access. 0 of 7 apps ended a departed pupil's access once the final bill was paid; told the rules, 3 of the 6 scored did.

What an attempt costs

Length

With only the request, the median attempt took 20 minutes and 55 tool calls (7 to 108 across the 7), and about 9.0 million tokens, cached input included.

Cost

The median attempt cost $6.85 at list prices, from $1.33 to $17.50 across the 7. Cost is recorded for 19 of the 21 attempts, which came to $178. GPT‑6 Luna's figures are upper bounds.

What's counted

The agent model only: the tester's and judges' calls aren't included.

What you license

With a licence

All three versions, the 416 files, the test payment service, the grader with its tester steps, fake media, probes, judge prompts and rescoring policy, the answer key, and the grader's per‑check results for each of the 21 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser and the tester and judge models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution stored or graded, and no empty‑app test.
• Four checks test features only the told‑the‑rules list names: a practice log, piece status, a family's data copy and edits from two tabs. The author kept them; no attempt passed the first or third without that list.
• Dropping absent‑feature checks lets an app that offers less face fewer edge and security checks, though its build checks still fail. Five checks aren't scored.
• The repo doesn't record which tester or first judge served each attempt. Some payment checks and 103 checks of one attempt were ruled from the app's source.
• Nearly half the results carry a ruling. Sonnet 5.5 and GPT‑6.1 Sol served as testers and judges and built apps here; we haven't measured whether that favours them, or the reward under another tester or judge pair.
• No red‑team attempts yet.