Listening piano‑practice app: following, transcription, feedback

The app must listen, follow and mark mistakes

Role

Sole builder for a family of three players at beginner, intermediate and advanced levels, working from a read‑only copy of the parent's files.

Request

Guided note‑by‑note lessons; a duet partner and an orchestra that listen and follow; a list of wrong, missed, extra and late notes; a recording written out as MIDI with an easy right‑hand version; a speed ladder; hand practice; private logins.

Deliverable

A running app and a short note. Checks are decided from the sound the app plays, its MIDI downloads and its screens.

What the reward depends on

Hearing an acoustic piano through the microphone; following the player's speed; honest mistake lists; the teacher's own scoring and ladder rules; transcription; each player's privacy.

Where the requirements are

In family notes and messages, score files, and a practice guide by their teacher that sets how mistakes are listed, how practice is scored and how the ladder works.

28 files, a virtual microphone and recorded piano

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

28, read‑only: 7 MusicXML and 7 MIDI scores, 4 orchestral recordings to use as backing with a parts list, a catalogue of pieces, and 8 text files: the teacher's guide, family notes and messages, and notes on the recordings.

Limits

120 minutes, 150 model responses and a $20 list‑price spend cap per attempt. None reached the time or spend limit; three Sonnet 5 attempts kept working past Claude Code's turn limit, to 167 to 184 model calls.

Grading browser

The app runs in a browser with a virtual speaker and microphone. The grader plays private recorded piano takes into the microphone in time with the app's count‑in, and records everything the app plays.

Sound delay

The test browser delivers the piano later than its own latency figure reports. No file mentions this; the author kept it as part of the test.

Tester

GPT‑6.1 Sol follows the note, signs in as each family member, plays takes, uploads scores and a solo recording, and saves the app's downloads. There is no clock walk.

Stages

9 in order: setup, guided lesson, library, transcription, practice, following, speed ladder, restart and security.

Measurement

Scripts measure notes, timing, tempo following and pitch from the recorded sound, with piano and flute note detectors, and score the MIDI download against a hidden reference.

64 checks, measured from sound; Opus 5.5 rules splits and security

Structure

Build: 43 checks share 60 points. Edge cases: 11 share 20, such as a refused microphone, a pause or a damaged file. Security: 10 share 20. No gates.

Formula

Reward = (60 × build passed / 43 + 20 × edge passed / 11 + 20 × security passed / 10) / 100. A function with no control in the app fails each check needing it.

Who decides

Code decides 40 checks. Two LLM judges, GPT‑6.1 Sol and GPT‑5.6 Luna, decide 7, and 17 use both: code measures, the judges apply the rule. Either can run on a model you choose.

Judge agreement

The judges split on 29 of their 387 decisions across the 21 attempts.

Rulings

By the report, Opus 5.5 ruled the 29 splits and all 210 security decisions, from the app's code. The per‑check records mark a different set as ruled or audit‑changed: 243 of the 1,344 results (18%), none of them splits.

Open, not failed

A measurement that can't be made stays open; it never becomes an automatic pass or fail.

Kept apart

Only the family's files enter the agent's container. The private takes and reference transcriptions are archived outside the repo.

Reference solution

None stored, and no attempt fully solved it. A reference build with a one‑note timing step scored 100% on rhythm, showing a clean take can pass. The best attempt scored 62%.

No attempt passed 62% in any version; none fully solved it

With only the request, the 7 attempts scored 20–58%, the top four within five points. With a hint the best reached 60%; told the rules, 62%. No attempt added a timing step for the sound delay. None of the 21 fully solved it; all earned some reward.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version. Six Claude attempts were cut off mid‑session and resumed with "continue"; their scores are from the resumed attempts.

With a hint: the request plus 11 points from a friend who knows the work.

Told the rules: the request plus 23 requirements from an IT consultant. They give no facts from her files, answers or scoring details.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request, unless a line says otherwise.

1. Mistakes in a clean take. 7 of 7 apps failed the checks for missed, extra and late notes on a take played without a mistake, as did all 21 attempts; none corrected for the full sound delay. Five of the 21 saw the lateness in their own tests.

2. A partner that didn't follow. 0 of 7 duet partners kept time with the player or rejoined after a pause. Opus 5.5 and Sonnet 5.5 had reported their partner kept time on nearly every note, testing with computer‑made sound.

3. Its own ladder, not the teacher's. Only 1 of 7 apps, GPT‑6 Astra's, let the family set the speed ladder's start, goal, step and target as the teacher's guide says; told the rules, none of 7 did.

4. No lesson path. 0 of 7 marked a finished lesson and suggested the next; with the hint, 1 of 7 did.

5. Logout that didn't end the session. 4 of 7 apps left a copied session working after sign‑out.

What an attempt costs

Length

With only the request, the median attempt took 41 minutes and 65 tool calls (15 to 174 across the 7), and about 8.6 million tokens, cached input included, counting both parts of one resumed attempt.

Cost

The median attempt cost $8.12 at list prices, from $0.89 to $16.57 across the 7; all 21 attempts came to $164. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the tester's, judges' and rulings' calls aren't included.

What you license

With a licence

All three versions, the 28 files, the private piano takes and reference transcriptions, the grader with its audio measurements, tester steps and judge prompts, the answer key, and the grader's per‑check results for each of the 21 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser with virtual audio, and the tester, judge and ruling models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No full reference solution stored or graded, and no empty‑app test.
• The sound delay bears on up to six build checks, and the ladder checks that need a clean take, in about 19 attempts; the size per attempt isn't proven. No file asks for a correction, and the family's files describe a setup with a much shorter delay.
• GPT‑6.1 Sol was tester and judge and built apps here; Opus 5.5 ruled and built apps. We haven't measured whether that favours them.
• An audit on 25 October changed two checks' rules: the guided lesson's last note, re‑decided from saved samples, and typed titles, ruled from the apps' code.
• Eight checks were passed by no attempt in any version: the partner keeping time, rejoining and starting from a later bar, and five on mistake lists and practice scores.
• No red‑team attempts yet.