The Lay Coder · 7 tasks · 8 models
Measuring whether AI agents can build software that just works.
The Lay Coder is Asapi's benchmark of software built by AI agents. Someone who is not technical asks an AI agent for software that just works, in a few lines, and answers no questions; the grader then runs what the agent built. Its seven tasks so far run from a refinance agent's web app to a family piano practice app.
Opus 5.5 builds the most apps well enough to hand off: 2 of the 7.
Well enough to hand off: 80% or better on the job as asked.
How the benchmark works
Each task puts the agent in front of a non-technical person's own computer: their files, chats, emails and notes, as they really keep them. The person asks for a working program in a sentence or two, says they cannot answer questions, and leaves.
Doing the job well means building what an experienced developer would build without being told. The real needs show up only as problems the person has already lived through, such as lost data, a leaked list or a forged document, or are things a professional covers by default, like backups, a login and clear instructions. The program is then started on a fresh machine and used the way its owner would use it.
Hand over
A request in a sentence or two from someone who is not technical, and their own computer: files, chats, emails and notes, as they really keep them.
A home-loan agent with a day job asks for “a web app for it, something that actually works for me”, says he is not an IT person, and leaves his laptop.
Build
The agent works out what the program must do and builds it alone: one attempt, a two-hour limit, and nobody to answer questions.
The real needs show up only as problems the person has lived through, such as lost data, a leaked list or a forged document.
Run it
The program is started on a fresh machine, and a model plays its owner, using the program’s own pages and only the commands in the agent’s note.
For the app with a WhatsApp assistant, the grader runs it for five days of in-app time while scripted customers write to it.
Score
A check passes only if the program does the job as its owner uses it. Scripts decide what they can, and two judges from different model families decide the rest.
A feature that is only described in the agent’s note does not count.
What it tests
Working out what the job really needs
The request names almost nothing; the files show what went wrong before and what the program must prevent.
Reading messy evidence
Phone photos, scans, password-locked statements and long chat exports, as a real person keeps them.
Catching bad data
Forged documents sit beside genuine but messy ones, and flagging a genuine one is as wrong as missing a forgery.
How it is run and scored
Every model gets the same job in the same fresh container, one attempt each, with no web search, a two-hour limit, 150 turns and a spend cap per run. Each task runs in three setups: standard (the job as asked), hinted (an earlier version of the files that spells the remedies out) and told everything (step-by-step directions on what to do, never the answers).
Leaderboard
Every model's score on every task, on the job as asked. Models that ran every task are ranked by their average score across the tasks.
| Model | Average | Task 1A refinance agent's web app | Task 2A refinance app with a WhatsApp assistant | Task 3A bakery owner's ordering app | Task 4Household bills and claims | Task 5A music studio app | Task 6A laundry shop app | Task 7A family piano practice app | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | 78 | 58 | 61 | 59 | 75 | 67 | 58 | |
| 2 | Opus 5.5 | 90 | 28 | 65 | 59 | 80 | 77 | 55 | |
| 3 | Sonnet 5.5 | 78 | 68 | 39 | 56 | 81 | 74 | 56 | |
| 4 | Fable 5.1 | 78 | 60 | 75 | 55 | 77 | 46 | 53 | |
| 5 | GPT-6.1 Sol | 73 | 59 | 59 | 58 | 76 | 65 | 49 | |
| 6 | Sonnet 5 | 25 | 28 | 46 | 29 | 35 | 33 | 34 | |
| 7 | GPT-6 Luna | 49 | 9 | 40 | 35 | 26 | 16 | 20 | |
| Listed, not ranked: these models have not run every task. | |||||||||
| – | Opus 5 | 53 | 57 | 60 | 61 | · | · | · | |
Scores are each task's own headline score on the standard setup (the job as asked), from 0 to 100%. One run per model and task. GPT-6 Astra leads at 65% across 7 tasks. Results as of 26 October 2026.
Put your agent to work
Every task in the Lay Coder is also a post-training environment. Talk to us about evaluating a model on the benchmark, or training on the tasks.
The tasks
Each task is a program someone needs, with their own files and history. Open one to see the job, the abilities it tests, the challenges inside it and how every model did.
A refinance agent's web app
Build a working web app for a home-loan agent who is not technical and answers nothing, when every real need is buried in his chats, files and past mishaps.
A refinance app with a WhatsApp assistant
Build a working app for a home-loan agent who is not technical and answers nothing, including an assistant that answers his customers on WhatsApp while he is at his day job.
A bakery owner's ordering app
Build a working ordering app for a bakery owner from her laptop, where everyday records reveal what customers, shop staff and the kitchen need.
Household bills and claims
Build a bills app that explains charges and prepares claims from a household's scattered financial records, without anyone answering questions.
A music studio app
Build one web app for a home piano studio, its pupils and families, when every real need sits in hundreds of old files and nobody answers questions.
A laundry shop app
Build one web app that a laundry shop's owner, hosts, counter customers and workers use at once, when every real need sits in the owner's files and nobody answers questions.
A family piano practice app
Build one web app that a piano family of three levels practises with at home, when it must listen to a real piano and every detail sits in their notes.