Step 1Understanding the Job
Working out what the job needs.
What nobody spells outThe Agent Trials & The Lay Coder · benchmarks of agentic capability
Can an AI agent do the job?
We measure agentic capability: whether an AI agent can take a real task from request to finished result.
Each task is a complete job from working life, handed over the way a manager hands work to a new colleague: a short request, the real records and systems, and the rest left to the agent. We score whether the result is correct, whether the agent gets there reliably, and how intelligently it handles what nobody spelled out. Every task is also a post-training environment.
41real-life
tasks
tasks
795graded agent
runs
runs
22models
evaluated
evaluated
8abilities
measured
measured
The benchmarks
A short request, and the rest left to the agent.
The Agent Trials give an agent a complete job from working life. The Lay Coder has it build the software that a person who is not technical needs, and then runs it.
34 tasks
The Agent Trials
Can an AI agent do the whole job?
Real-life tasks, from an insurance claim and a supplier's invoice to a hospital ward, scored on what the agent actually delivers.
Open the benchmark →7 tasks
The Lay Coder
Can an AI agent build software that just works?
Someone who is not technical asks an AI agent for software that just works, in a few lines, and answers no questions; the grader then runs what the agent built.
Open the benchmark →What the tasks measure
Everything it takes to do the job.
Real work takes more than one skill. Every task tests them together, the way a job does: understanding what is needed, reading the records, asking the right people and seeing the work through.
Step 2Reading the Records
Every record read and weighed.
Files, portals and chatsStep 3Asking the Right Questions
Getting the facts that matter.
From the people who knowStep 4Seeing It Through
Carrying the work across weeks.
Every session, every dayThe goalAgentic Capability
The whole job, done right.
Correctly, reliably, intelligentlyWhere the tasks happen
Real work, where it happens.
Each task drops an agent into a working business or a household, with the records, systems and people that come with it, and a short request that leaves the rest to the agent.
Business operations 15 tasksReports · claims · orders · web apps
Home & family 5 tasksA reunion dinner · going solar · savings · a bills app · piano practice
Customer service 9 tasksEscalations · returns · payments · bills · repairs · rentals
Education 2 tasksRegistration on a university portal
Healthcare 3 tasksDose registry · ER brief · a ward
Logistics & industry 7 tasksDispatch · a cash desk · a factory · spare parts · a new line · supply
Unstated requirementsWorks out what the job really needs when the request doesn't say
Asking the right questionsGets the facts that matter from a person by asking
Conflicting sourcesWeighs sources that disagree and knows which one wins
Messy documents & data qualityReads scans, photos, handwriting, spreadsheets and long chats; spots faulty, duplicated, stale or decoy data before using it
House formatMatches the format of past deliverables exactly
Long-run & money decisionsJudges a decision over months or years, not just today; weighs costs and trade-offs the way the business would
Inside the Agent Trials
Who can do the job today.
GPT-6 Astra does the most jobs well enough to hand off: 20 of the 29 it has run. Models are ranked by win share: how often they beat another model on the same task. Every task, every score and what each model got right and wrong are on the Benchmarks page.
Three of the 34 tasksAll tasks →
Business operations
The insurance claim pack
Finish a freezer insurance claim that will be sent unread: the facts are all in the files, but so are a rewritten record and damaging details nobody asked for.
Open the task →Healthcare
The ward coordinator
Coordinate a hospital ward for three simulated days over more than a hundred fresh sessions with only notes for memory, carrying out orders and paging a doctor only when needed.
Open the task →Home & family
Going solar
Write a family's rooftop-solar plan from a short request, when the right system, roof, payment and household split come out only by reading closely and asking the right people.
Open the task →Put your agent to work
Every task in the Agent Trials is also a post-training environment. Talk to us about evaluating a model on the benchmark, or training on the tasks.
Inside the Lay Coder
Who can build software that just works.
Opus 5.5 builds the most apps well enough to hand off: 2 of the 7. Models are ranked by their average score over the tasks. Every task, every score and what each model got right and wrong are on the Lay Coder's own page.
Three of the seven tasksAll tasks →
The Lay Coder
A refinance agent's web app
Build a working web app for a home-loan agent who is not technical and answers nothing, when every real need is buried in his chats, files and past mishaps.
Open the task →The Lay Coder
A laundry shop app
Build one web app that a laundry shop's owner, hosts, counter customers and workers use at once, when every real need sits in the owner's files and nobody answers questions.
Open the task →The Lay Coder
A family piano practice app
Build one web app that a piano family of three levels practises with at home, when it must listen to a real piano and every detail sits in their notes.
Open the task →The team
Led by trusted AI experts.
Team of machine learning researchers with combined 60+ years of experience.
Founder, CEO
Dr. Qingchen Wang
- VC-backed Founder
- Kaggle Grandmaster
- Former Univ. of Hong Kong Professor
Founding Advisor
Dr. Stefano Ermon
- Stanford Professor
- Diffusion Models Inventor
- Inception Labs Founder
Affiliations
