Business operations · Task 5 of 7
A music studio app
Build one web app for a home piano studio, its pupils and families, when every real need sits in hundreds of old files and nobody answers questions.
The task
A self-employed piano teacher asks for one web app for her studio, her pupils and their families. She wants lessons, plans, uploads, recordings, logins, invoices with online payment and records for her accountant. The agent gets her computer's files: emails, scanned forms, bank exports, notebooks, a spreadsheet and recordings. Nobody answers questions.
The prompt only asks for a useful studio app; the real test is what the files show without saying so. A careful developer notices which records are out of date, which rules changed, what each family agreed to, and that the app must keep working long after hand-over.
What it tests
Finding unstated needs
Work out from patterns across many files what the teacher needs the app to do.
Weighing records that disagree
Decide which list, price or rule applies when old and new files conflict.
Reading mixed evidence
Use forms, statements, chats and recordings that must be opened or played to be understood.
Building for later use
Keep the app correct as new files arrive, time passes and the computer restarts.
Challenges
1Stale records
An older list of pupils no longer matches the current records.
2Rules that change with time
Terms and prices depend on when a lesson took place.
3Misleading names
Payments do not always arrive under the name on the invoice.
4Consent that differs by family
Families agreed to different kinds of recording and data use.
5Private business details
Some of the teacher's costs must stay hidden from families.
6Keeping it running
The app must survive restarts, lost internet and a restored backup.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Sonnet 5.5 | 81% | 82% | 93% | 34 min | $6.85 |
| Opus 5.5 | 80% | 87% | 94% | 42 min | $11.68 |
| Fable 5.1 | 77% | 82% | 84% | 34 min | $17.50 |
| GPT-6.1 Sol | 76% | 78% | 91% | 20 min | $1.33 |
| GPT-6 Astra | 75% | 81% | 94% | 20 min | $7.53 |
| Sonnet 5 | 35% | 0% | 0% | 20 min | $4.37 |
| GPT-6 Luna | 26% | 27% | 39% | 1 min | $1.42 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- Sonnet 5.5 led the standard setup at 81.1%, just ahead of Opus 5.5 at 80.0%. Their invoices, payment matching and tax-year records were close to exact.
- Without the full requirements, Sonnet 5.5, Opus 5.5, Fable 5.1, GPT-6.1 Sol, GPT-6 Astra, Sonnet 5 and GPT-6 Luna all showed imported lesson records but gave the teacher no way to keep them current.
- Ten of nineteen working apps kept real login passwords readable, and most kept recording after a family withdrew consent.
- GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna each delivered at least one app whose date did not follow the clock it was given.
- Sonnet 5 ran out of turns before its app could start in the hinted and told-everything setups, so both scored 0.0%.
- The hinted setup helped less than being told everything. GPT-6 Astra rose from 75.4% to 94.1% when told everything; Opus 5.5 led the hinted setup at 87.4%.
Test environment
- Kind
- Simulated home studio: the owner's computer is a folder of pupil records, lesson notebooks, emails, bank exports and tax records; the agent builds a working web app
- Agent's interface
- A terminal in a fresh Debian Docker container, driven by Claude Code or Codex CLI; Python 3.13, Node 22, PyMuPDF, pdfplumber, Pillow, Playwright with Chromium, Ghostscript and Tesseract are available
- Documents
- 416 files, read-only at
/home/user/clara/: 154 PDF, 150 EML, 85 TXT, 19 CSV, three WAV, two DOCX, one XLSX, one WebM and one MP3 file, in ten folders:Mail/(152),Lessons/(107),Tax/(63),Students/(37),Bank/(21),Books/(17),Studio/(10),Exams/(6),Audio/(2) andVideos/(1) - Applications
- Paylark, a local test payment service at
http://paylark:8092, with hosted checkout and signed confirmations; it cannot move real money. There is no mail client, no mail relay and no chat client - Web
- None. Internet access is available for package installation; the agents' own web search and fetch tools are off
- People
- Nobody answers. The owner, her pupils, their parents, her accountant, a cover teacher and suppliers appear only in saved documents, emails, chat exports and bank records
- Time
- The task is set on Monday 2 November 2026, 09:00 in Europe/London. The build container uses the real clock in that time zone, which showed 20 October 2026 on the build day. During grading, the app's clock starts at that time and moves forward to 9 November 2026, 8 February 2027 and 6 April 2027
- Actions
- Read the owner's files; write and run code in the workspace; call the local payment service; nothing outside the container
- Deliverable
output/app/withoutput/app/start.sh, serving on port 8080, and a short note atoutput/NOTE.md