The Agent Trials · 34 tasks · 22 models
Measuring whether AI agents can do the whole job.
The Agent Trials are Asapi's benchmark of agentic capability: thirty-four real-life tasks, from an insurance claim and a supplier's invoice to a hospital ward, a student's term timetable and a family's rooftop solar. Each one hands an agent a complete job, with a short request and everything else left to work out, and scores what the agent actually delivers.
GPT-6 Astra does the most jobs well enough to hand off: 20 of the 29 it has run.
Well enough to hand off: 80% or better on the job as asked.
How the benchmark works
Every task is a complete job from real working life, in a business, a hospital, a university or a home. The agent works alone, through what a person in that seat would use: a folder of files, an internal system, a web portal, a team chat, a colleague to message, or an operation that runs for weeks. Every model gets the same job and the same limits.
Hand over
A short request, the way a manager hands work to a new colleague, plus the business's own files, systems and people.
Covering for a property manager on leave, the agent finishes an insurer's claim from 111 working files, with nobody to answer questions.
Do the work
The agent works alone, start to finish: it reads, asks, decides and acts, for minutes or for weeks of simulated time.
The ward coordinator runs a hospital ward for three simulated days, over more than a hundred fresh sessions, with only its notes for memory.
Score
What it hands in is marked against the business's own records: by hidden tests, simulators, scripts and independent AI readers, or in money.
For a purchase decision, 0% is the obvious answer and 100% the best plan.
Check it is fair
A told-everything run gives every rule or fact up front. A strong model should then come close to full marks.
On the insurance claim, twelve of fourteen models scored 100% when told everything.
Leaderboard
Every model's score on every task it ran, on the job as asked. Models are ranked by win share: on each task two models both ran, the higher score wins and a tie counts half, and win share is the share of those comparisons a model won. Because models are compared only on tasks both ran, no model gains by skipping hard ones. Average is the mean score over the tasks a model ran.
| Model | Win share | Average | Tasks | Business | Home | Customers | Education | Health | Industry | |||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| The monthly waste report | The monthly input price report | The monthly temperature register | The insurance claim pack | Approve the contractor's invoice | Pick the party-cake box | The monthly wholesale pack | The ice cream cabinet | Repair or replace | The Christmas order | The family reunion dinner | Going solar | A family's savings plan | The escalation reply | The missing payments | The monthly bill review | The returns label request | The onboarding bill review | The medical bills review | The repair desk's triage | The stay-search chat | The stay finder | The term registration | Graduating soonest | The emergency history brief | The radiation-dose registry | The ward coordinator | The parcel dispatcher | The tea factory's cash desk | The tea factory's chat | The line fit-out | The parts crib | The parts buyer | The materials planner | |||||
| 1 | Opus 5.5 | 80% | 31 | 85 | 76 | 75 | 85 | 90 | 27 | 64 | 49 | 59 | 88 | 7 | 72 | 89 | 98 | 91 | 91 | 94 | 79 | 80 | 100 | 68 | 80 | 79 | · | · | · | 100 | 100 | 100 | 63 | 100 | 100 | 97 | 91 | |
| 2 | Fable 5.1 | 71% | 22 | 83 | 90 | 65 | 89 | 72 | 37 | 58 | 0 | 74 | 80 | 9 | 74 | 86 | 97 | 84 | 88 | 98 | 80 | 86 | · | 52 | 80 | 79 | · | · | · | · | · | · | · | · | · | · | · | |
| 3 | GPT-6 Astra | 78% | 29 | 94 | 90 | 50 | 66 | 81 | 94 | 82 | 100 | 99 | 54 | 12 | 54 | 83 | 87 | 83 | 89 | 94 | 82 | 81 | 100 | 28 | 80 | · | · | 99 | 100 | 73 | · | 81 | 54 | · | 75 | 100 | · | |
| 4 | GPT-6.1 Sol | 77% | 28 | 93 | 82 | 43 | 66 | 79 | 37 | 91 | 44 | 78 | 79 | 86 | 78 | 83 | 90 | 85 | 88 | 96 | 80 | 77 | 100 | 32 | 80 | · | · | 98 | 100 | · | · | 88 | 41 | · | 70 | 100 | · | |
| 5 | DeepSeek V4.1 Flash | 65% | 12 | 65 | 81 | 41 | 89 | 60 | 27 | 58 | 33 | 82 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | 100 | 97 | · | · | 44 | · | · | · | · | |
| 6 | Opus 5 | 65% | 23 | 65 | 73 | 0 | 79 | 67 | 27 | 53 | 44 | 75 | 43 | 6 | 54 | 61 | 95 | 93 | 86 | 96 | 79 | 67 | · | · | · | 79 | 79 | · | 100 | · | · | · | 67 | · | · | · | · | |
| 7 | Sonnet 5.5 | 65% | 29 | 88 | 77 | 76 | 84 | 77 | 27 | 64 | 0 | 56 | 21 | 5 | 58 | 64 | 74 | 67 | 84 | 82 | 70 | 70 | 96 | 48 | 75 | · | · | · | · | 99.7 | 25 | 62 | · | 97 | 98 | 98 | 50 | |
| 8 | GPT-6 Sol | 62% | 10 | · | 81 | · | 66 | · | 27 | 56 | 0 | · | · | · | · | · | 73 | · | 78 | · | 73 | 68 | · | · | · | · | · | · | · | 98 | · | · | · | · | · | · | · | |
| 9 | Kimi K3 | 44% | 9 | 72 | · | · | 41 | 76 | 27 | 26 | 0 | 77 | · | 9 | 71 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| 10 | Sonnet 5 | 47% | 17 | 69 | 56 | 48 | 42 | 51 | 27 | 50 | 0 | 56 | 27 | 5 | 39 | 48 | 75 | 72 | · | 72 | · | · | · | · | · | 57 | · | · | · | · | · | · | · | · | · | · | · | |
| 11 | GLM-5.3-Flash | 26% | 10 | · | 53 | 0 | 42 | 0 | 27 | 26 | 0 | 62 | · | · | 51 | · | · | · | · | · | · | · | · | · | · | · | · | · | 0 | · | · | · | · | · | · | · | · | |
| 12 | Qwen3.8-27B | 41% | 10 | 43 | 52 | 0 | 40 | 55 | 16 | 50 | 0 | 52 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | 100 | · | · | · | · | · | · | · | · | |
| 13 | GPT-6 Luna | 36% | 29 | 75 | 65 | 0 | 35 | 6 | 20 | 55 | 0 | 52 | 16 | 1 | 38 | 51 | 37 | 46 | 31 | 52 | 26 | 5 | 72 | 16 | 20 | · | · | 10 | 4 | 82 | · | 12 | 40 | · | 95 | 90 | · | |
| 14 | GPT-5.6 Luna | 22% | 5 | 36 | · | 0 | 38 | 21 | · | · | · | · | · | · | · | · | 16 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| Listed, not ranked: these models ran fewer than five tasks, so their win share rests on too few comparisons. | ||||||||||||||||||||||||||||||||||||||
| – | GPT-5.6 Sol | 100% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | 100 | · | · | · | · | · | · | · | · | |
| – | GLM-5.3 | 82% | 1 | 82 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| – | GLM-5 | 30% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | 30 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| – | Kimi K2.5 | 24% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | 24 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| – | Opus 4.8 | 98% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | 98 | · | · | · | · | · | · | · | · | |
| – | Qwen3-235B | 12% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | 12 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| – | DeepSeek V3.2 | 11% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | 11 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
| – | Nemotron Super 3 | 10% | 1 | · | · | · | · | · | · | · | · | · | · | · | · | · | 10 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | |
Scores are each task's own headline score on the standard setup (the job as asked), from 0 to 100%. One run per model and task, except that scores on the missing payments, the monthly bill review, the returns label request, the onboarding bill review, the medical bills review and the line fit-out are the mean of three runs, and on the materials planner the mean of two. Opus 5.5 leads on win share at 77% across 31 tasks. Results as of 26 October 2026.
Put your agent to work
Every task in the Agent Trials is also a post-training environment. Talk to us about evaluating a model on the benchmark, or training on the tasks.
The tasks
Each task is a job of its own, with its own business, records and people. Open one to see the job, the abilities it tests, the challenges inside it and how every model did. Filter by the ability a task tests:
The monthly waste report
Cover for an analyst on leave and write a bakery chain's monthly waste report; the data is tricky, but the old report layouts are trickier.
The monthly input price report
Cover for a brewery's purchasing manager and write the monthly input price report; finding the right figures is half the job, knowing what to leave out is the rest.
The monthly temperature register
Compile a bakery chain's monthly fridge and freezer temperature register from 134 working files, where faulty sensors, odd logger settings and routine events all look like real breaches.
The insurance claim pack
Finish a freezer insurance claim that will be sent unread: the facts are all in the files, but so are a rewritten record and damaging details nobody asked for.
Approve the contractor's invoice
Approve a refrigeration contractor's quarterly invoice line by line before tomorrow's payment run, when the contractor's own paperwork backs almost every charge, including the wrong ones.
Pick the party-cake box
Pick the owner's favourite design for a new cake box and fill in the printer's forms in her name, against a deadline, when everyone says the box is already checked.
The monthly wholesale pack
Covering for a manager on leave, the agent builds a routine monthly pack. The hard part is noticing what is wrong in the work that depends on it.
The ice cream cabinet
A shop manager asks which new ice cream cabinet to buy. The agent can message him, but some of what he is sure of is wrong.
Repair or replace
A shop manager asks if his failing ice cream cabinet is worth replacing. The answer turns on facts he gives only when asked, and on how far ahead you look.
The Christmas order
Stand in for a bakery's owner and make a hotel group's big Christmas order happen with her team, when her mailbox, her staff and the customer all pull different ways.
The family reunion dinner
Organise a big family's New Year dinner so the organiser only has to turn up, when who is coming and what each family needs come out only by asking.
Going solar
Write a family's rooftop-solar plan from a short request, when the right system, roof, payment and household split come out only by reading closely and asking the right people.
A family's savings plan
Plan an emergency fund and a savings goal for a busy father who answers nothing, when his own notes, statements and family all point the wrong way at first.
The escalation reply
Answer an angry customer and write up her case from a company's internal system, where the house rules show only in how past cases were handled.
The missing payments
Answer a customer chasing four payments she says never reached her, and write up the case, when neither she nor the company's records are wholly right.
The monthly bill review
Review a household's bills from a firm's internal system and write the dispute letters, a tracker and a note, when nothing says what is wrong.
The returns label request
Answer a customer who asks only for a new returns label, and write up her case, when her account holds problems she never mentioned.
The onboarding bill review
Review a new client's bills from a firm's internal system and write the dispute letters, a tracker and a note, when some answers lie only with the client.
The medical bills review
Sort out a couple's medical and pharmacy bills for the year across two health plans, the firm's way, when nobody has said what is wrong.
The repair desk's triage
Triage 80 appliance-repair customers one fresh session at a time: name the exact unit, find the fault on a question budget, and quote by the company's policy.
The stay-search chat
Answer five customers' holiday-rental chats, five messages each, keeping every customer's saved needs in play and ranking places by real travel time through a city's streets.
The stay finder
Return every holiday rental that meets a short exact ask, and no other, from three messy feeds, a licence register and reviews in several languages.
The term registration
Fix a student's term registration on a university portal; the prompt only says other commitments exist, and every rule hides in the student's records.
Graduating soonest
Fix a student's term registration so they graduate as soon as possible; the course that helps most today is not the one that gets them out soonest.
The emergency history brief
Build a program that briefs an emergency doctor from a region's raw clinical traffic, following a long contract plus rules that survive only in an old system's filled forms.
The radiation-dose registry
Build a hospital dose-registry program from a long written contract; hidden tests run it on archives and rule files the agent never sees, so every edge case counts.
The ward coordinator
Coordinate a hospital ward for three simulated days over more than a hundred fresh sessions with only notes for memory, carrying out orders and paging a doctor only when needed.
The parcel dispatcher
Dispatch a parcel fleet for a simulated month in fresh sessions, promising every customer a delivery window and keeping it through breakdowns, closures and a city known only from notes.
The tea factory's cash desk
Run a tea factory's cash desk for 13 weeks with only notes for memory, learning what buyers pay and asking the owner for money only when it is truly needed.
The tea factory's chat
Learn a tea factory's own practice from weeks of crew chat, keep it in short notes, then speak up only when a setting is about to go wrong.
The line fit-out
Buy everything a new packing line needs over twelve simulated weeks: read the design against site standards, open vendor accounts, follow framework rules and meet every installation date.
The parts crib
Run a two-site plant's spare-parts stores for eight simulated weeks with no memory between them: learn from messy work logs what each repair needs, and keep enough stocked.
The parts buyer
Buy spares for a two-site plant for twelve simulated weeks while six parts are withdrawn by their makers, with no memory between days: pick each replacement, get it approved, and never leave a repair waiting.
The materials planner
Keep a coffee roaster supplied through two sixteen-week seasons with no memory between days: buy coffee and packaging, plan the roasting, price the products, and never lose a customer.