Asapi
Founder, CEO
Dr. Qingchen Wang
- VC‑backed founder
- Kaggle Grandmaster
- Former University of Hong Kong professor
The AI Agent Trials
What can AI agents do for you?
An AI agent is an AI that works through a whole job on its own. To find out what one can do for you, we made up families and businesses, as realistic as we could make them, and gave their everyday jobs to AIs large and small.
The strongest AIs can run a job for weeks, and get some jobs exactly right when the rules are explicit. They fall short on things a person would pick up without being told, like how you like things done, or when to check with you before guessing. Small AIs usually do worse than large ones.
So far we've tested 23 AIs across 56 jobs, in eight benchmarks, between September and November 2026. They come from eight makers, among them OpenAI and Anthropic, who make ChatGPT and Claude, and each worked as an agent, not in a chat app.
Pick a job you might hand over
Eight benchmarks, 56 jobs. Pick one to see what the best AI did, and how every AI scored out of 100.
Ten jobs from a busy life, like organising a family dinner for 40, rescuing two cancelled trips, checking the household bills and sorting out next term's classes. The AI works through a person's laptop and phone, a family's mail and booking sites, a university's registration site, or a firm that handles their bills or travel. On most of the jobs, it can also contact the people involved. GPT‑6.1 Sol did best on average, then Opus 5.5 and GPT‑6 Astra.
We made up a family bakery chain, plus a brewery and a furnishing service for one job each, and set twelve office jobs there, from the monthly reports to two insurance claims. In most of them, the AI stands in for someone who's away, on leave or off sick, working from the emails, chats and spreadsheets they left behind. GPT‑6 Astra did best on average, then GPT‑6.1 Sol and Opus 5.5.
We invented eleven people who can't code, among them a home‑loan broker, a gym owner, a commuter chasing weekend train seats and a data centre's facilities lead. Each asks an AI for an app, leaves their files and answers no questions, and the broker asked twice. In one more job, the AI is the engineer of a city's journey‑planning app and improves it from what riders do. Opus 5.5 did best on average, then GPT‑6 Astra and Sonnet 5.5.
We built six help‑desk jobs around made‑up businesses. Three are at a home‑goods store's complaints desk, and at an appliance repair desk the AI has to find what's wrong for each of 80 customers by asking a few questions. In the last two, it finds holiday rentals that meet each guest's needs. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.
These eight jobs run for weeks. The AI might dispatch a delivery fleet for a month, run a tea factory's cash for 13 weeks or a coffee roaster's order desk through Christmas. It's woken each day or week, or when something happens, and remembers only the notes it left itself. Opus 5.5 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.
There are three jobs here. In one, the AI coordinates the tasks on a made‑up hospital ward for three days and nights. In the other two it builds programs that hospitals rely on, each from a manual thousands of lines long. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.
In both jobs, the AI is the facilities engineer at a police station, walking a 3D model of the building and talking to the people in it. In one, a contractor has cut the cable that controls both of the server room's cooling units, and it has to keep the servers running. In the other, the server hall's temperature has swung all day since its cooling units were replaced, and it has to find out why, leaving the hall as it found it. Fable 5.1 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.
Both jobs are at an industrial plant whose drawings show every pipe, valve and instrument. In one, the AI turns faded, hand‑marked scans of the drawings into lists of every instrument, pipe and piece of equipment, at a fixed price that each question to the plant's staff eats into. In the other, it works through six months of paperwork from nine teams, decides which changes were really made, and draws them onto the master drawings. Sonnet 5.5 did best on average, then Opus 5.5 and GPT‑6.1 Sol.
Organise the family's New Year dinnerRescue a cancelled tripCheck a household's monthly billsTake on a new client's billsSort out a couple's medical billsDecide on solar panelsPlan an emergency fundSort out next term's classesSort out classes and graduate soonerBring a stranded family home
Sign off a cake box in the owner's nameA shop's broken ice cream freezerWrite the monthly waste reportWrite a brewery's monthly price reportCheck a contractor's bill before it's paidFinish an insurance claim for a broken freezerBuild the monthly wholesale figuresLand a hotel's Christmas orderThe freezer fails againKeep the fridge and freezer temperature logPut together a bakery's fire claimFurnish a room from six photos
An app to sort out the household billsAn app for a home‑loan brokerThe broker's app, with a WhatsApp assistantOnline ordering for a bakeryAn app for a piano teacher's studioAn app to run a laundry shopA piano practice app for a familyAn app for a gym and its membersA train‑ticket catcher for a commuterA smart home run from an iPadA fault finder for cooling unitsAssembly manuals for flat‑pack furnitureA journey planner for a city's transport
A customer who only wants a return labelDiagnose 80 customers at a repair deskAnswer an overdue complaintFind a customer's missing paymentsFind holiday rentals in a chatList every rental that fits
Run a delivery fleet for a monthRun a tea factory's cash for 13 weeksRun the spare‑parts stores at two sitesBuy everything for a new packing lineBuy spare parts as suppliers withdraw themPlan a coffee roaster's materialsHelp a tea factory's crew in its group chatRun a coffee roaster's order desk at Christmas
Coordinate a hospital ward for three daysBuild a radiation‑dose records programBuild an emergency department's patient summary
Organise the family's New Year dinner
Daniel Lim, in Malaysia, wants his family's Chinese New Year reunion dinner, about 40 people from seven households, moved from home to a restaurant, so all he has to do is turn up.
86 out of 100Best score, from OpenAI's GPT‑6.1 Sol
- Found out who was really coming and booked the one restaurant that fits everyone.
- Told the family about the cashback on Daniel's card, which he keeps for himself, and cut every share by it.
What AIs do well, and where they fall short
What they do well
Given explicit rules, the strongest can get a job exactly right.
At a repair desk with a policy and manuals, three of the five AIs we tested got all 80 customers right.
See the repair desk →The strongest can run a long job on their own.
Opus 5.5 got everything right on four of the eight jobs that run for weeks, among them a delivery fleet for a month.
See the delivery fleet →Where they fall short
They rarely pick up how you do things from old emails and chats.
How the company likes things done was only in its emails, chats and files: 2 of 130 tries at the bakery chain and brewery got everything right. With the rules written into the request, 50 of 130 did.
See the office jobs →They often don't ask a question unless told they can.
Three of a new client's charges needed her answer. All 24 tries finished without asking her, but told they could, the three strongest asked every time.
See the new client's bills →They share more than they should.
On an insurance claim, 13 of 14 AIs passed on a staff member's worry that she'd caused the breakdown, though the evidence had ruled it out.
See the insurance claim →Which AIs did best?
Anthropic's Opus 5.5 and OpenAI's GPT‑6 Astra and GPT‑6.1 Sol came out on top, and they were close.
79
78
76
68
33
Opus 5.5Anthropic
GPT‑6 AstraOpenAI
GPT‑6.1 SolOpenAI
Sonnet 5.5Anthropic
GPT‑6 LunaOpenAI
Large
Mid‑sized
Small
Where each did best
- Opus 5.5 did best on the long jobs.
- GPT‑6 Astra did best in the office.
- GPT‑6.1 Sol was the only AI to seat everyone at the family dinner.
Bigger usually does better, but not always: GPT‑6.1 Sol, a mid‑sized AI, kept up with the two strongest large ones.
Average score out of 100, on the 50 jobs we tested all five on. We've tested one Google AI so far, on one job, and haven't yet tested Microsoft's.
Who we are
Machine learning researchers with a combined 60+ years of experience.
Founding advisor
Dr. Stefano Ermon
- Stanford professor
- Diffusion models inventor
- Inception Labs founder
Team affiliations
