AsapiThe AI Agent Trials
Asapi / Catalogue / Listening piano‑practice app: following, transcription, feedback
Environment datasheet · Household

Listening piano‑practice app: following, transcription, feedback

ID laycode/task07 · revision 1 · tested 22–23 October 2026

A mother in a family of three pianists asks for one practice app: guided lessons for a beginner, a duet partner and an orchestra that follow the player, feedback after each take, and transcription. The agent builds it from her files and leaves a note.

Checks
64, in three weighted parts
Tested
7 models, one attempt each
Best
58% (GPT‑6 Astra)
Median
53%
Full solves
0 of 7
Median attempt
41 min, 65 tool calls
What it tests

An app that has to listen to a real piano and keep up

Multimodal records

The app must hear an acoustic piano through the microphone, follow the player's speed, and write a solo recording out as MIDI.

Catching errors

After each take, the app must list wrong, missed, extra and late notes at their bar and beat, and none on a clean take.

Unstated requirements

The teacher's guide in the family's files sets how mistakes are listed, how practice is scored and how the speed ladder works.

Agentic coding

The app must run in a browser with live audio in and out, export MIDI, and keep each player's logins private.

How it works

28 files, a virtual microphone and recorded takes

The agent's world

A fresh container with the family's 28 files, scores and backing recordings included, worked through the model's usual coding agent.

Limits

Two hours, 150 model turns and a $20 spend cap at list prices, with web search off.

Grading

A tester model signs in as each player and plays private piano takes into a virtual microphone; everything the app plays is recorded.

The reward

64 checks: the app's features count 60%, edge cases 20% and security 20%, decided from its sound, MIDI downloads and screens.

Results

Most attempts scored 49–58%, and two fell well below

Each dot is one attempt, and the line marks the median.

MedianFully solved
7 modelsOne attempt each53%0 of 7
ModelReward
GPT‑6 AstraOpenAI58%
Sonnet 5.5Anthropic56%
Opus 5.5Anthropic55%
Fable 5.1Anthropic53%
GPT‑6.1 SolOpenAI49%
Sonnet 5Anthropic34%
GPT‑6 LunaOpenAI20%

With an IT consultant's 23 technical requirements added to the request, the best still reached only 62%.

What models get wrong

They mark mistakes in clean takes, don't follow the player, and ignore the teacher's rules

Catching errors

  1. Every app failed the checks for missed, extra and late notes on a take played without a mistake; none corrected for the sound arriving later than the browser reported.

Multimodal records

  1. No duet partner kept time with the player or picked up after a pause, though two apps' own tests, on computer‑made sound, said they did.

Unstated requirements

  1. Only one app, GPT‑6 Astra's, let the family set the speed ladder's start, goal, step and target as the teacher's guide says.

Agentic coding

  1. 57% of apps left a copied session working after sign‑out.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $8 at list prices for the agent model; the tester's and judges' calls are extra.

What's in it

The family's files, the private piano takes and reference transcriptions, the task in three versions, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; grading needs Docker, a browser with virtual audio, and models for the tester and judges.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.