Asapi
Founder, CEO
Dr. Qingchen Wang
- VC‑backed founder
- Kaggle Grandmaster
- Former University of Hong Kong professor
The AI Agent Trials
What can AI agents do for you?
An AI agent is an AI that works through a whole job on its own. To find out what one can do for you, we made up families and businesses, as realistic as we could make them, and gave their everyday jobs to AIs large and small.
The strongest AIs can run a job for weeks, and get some jobs exactly right when the rules are explicit. They fall short on things a person would pick up without being told, like how you like things done, or when to check with you before guessing. Small AIs usually do worse than large ones.
So far we've tested 28 AIs across 77 jobs, in ten benchmarks, between September and November 2026. They come from eight makers, among them OpenAI and Anthropic, who make ChatGPT and Claude, and each worked as an agent, not in a chat app.
Pick a job you might hand over
Ten benchmarks, 77 jobs. Pick one to see what the best AI did, and how every AI scored out of 100.
Ten jobs from a busy life, like organising a family dinner for 40, rescuing two cancelled trips, checking the household bills and sorting out next term's classes. The AI works through a person's laptop and phone, a family's mail and booking sites, a university's registration site, or a firm that handles their bills or travel. On most of the jobs, it can also contact the people involved. GPT‑6 Astra did best on average, then GPT‑6.1 Sol and Opus 5.5.
We made up a family bakery chain, plus a brewery for one job, and set eleven office jobs there, from the monthly reports to two insurance claims. In most of them, the AI stands in for someone who's away, on leave or off sick, working from the emails, chats and spreadsheets they left behind. GPT‑6 Astra did best on average, then GPT‑6.1 Sol and Opus 5.5.
We invented eleven people who can't code, among them a home‑loan broker, a gym owner, a commuter chasing weekend train seats and a data centre's facilities lead. Each asks an AI for an app, leaves their files and answers no questions, and the broker asked twice. In one more job, the AI is the engineer of a city's journey‑planning app and improves it from what riders do. Opus 5.5 did best on average, then GPT‑6 Astra and Sonnet 5.5.
We built seven help‑desk jobs around made‑up businesses. Three are at a home‑goods store's complaints desk, one is a furniture store's warranty claim that rests on a customer's photos, and at an appliance repair desk the AI has to find what's wrong for each of 80 customers by asking a few questions. In the last two, it finds holiday rentals that meet each guest's needs. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.
Eight of these nine jobs run for weeks. The AI might dispatch a delivery fleet for a month, run a tea factory's cash for 13 weeks or a coffee roaster's order desk through Christmas. It's woken each day or week, or when something happens, and remembers only the notes it left itself. In the ninth, it designs rooftop solar and a battery for a tea factory. Opus 5.5 did best on average, then GPT‑6.1 Sol and GPT‑6 Astra.
There are three jobs here. In one, the AI coordinates the tasks on a made‑up hospital ward for three days and nights. In the other two it builds programs that hospitals rely on, each from a manual thousands of lines long. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.
In two of the three jobs, the AI is the facilities engineer at a police station, walking a 3D model of the building and talking to the people in it: a contractor has cut the cable that controls both server‑room cooling units, or the server hall's temperature swings all day. In the third, it's a rigging engineer, proving how a crew of two can move an 800 kg unit up a ramp onto its plinth in a hospital's new plant room. Fable 5.1 did best on average, then GPT‑6 Astra and Opus 5.5.
All four jobs are at process plants, whose drawings show every pipe, valve and instrument. The AI turns faded scans, and in another job photographed pencil sketches, into lists of every instrument, pipe and piece of equipment, paying for each question to the plant's staff. It also brings a plant's master drawings up to date from six months of paperwork, and leaves a watchdog program in charge of a running plant for a day. GPT‑6 Astra and GPT‑6.1 Sol did best on average, then Sonnet 5.5.
Six jobs around people's homes. The AI furnishes a living room from six photos, replaces a flooded family's furniture and gets a house ready for a family of six. It also plans a homeowner's bedroom lighting, fixes a wet condo floor overnight with what's in the condo, and designs and quotes rooftop solar for a row house. Opus 5.5 did best on average, then Sonnet 5.5 and Fable 5.1.
Eleven jobs with two robot arms in a physics simulation, where a loose grip lets things slip. In ten, the AI moves and fixes boards, from setting down a beam to building a dog house's frame and roof. In the eleventh, it swings a bat at ten pitches, seeing only what a batter would see. GPT‑6.1 Sol is the only AI tested on most of them.
Organise the family's New Year dinnerRescue a cancelled tripCheck a household's monthly billsTake on a new client's billsSort out a couple's medical billsDecide on solar panelsPlan an emergency fundSort out next term's classesSort out classes and graduate soonerBring a stranded family home
Sign off a cake box in the owner's nameA shop's broken ice cream freezerWrite the monthly waste reportWrite a brewery's monthly price reportCheck a contractor's bill before it's paidFinish an insurance claim for a broken freezerBuild the monthly wholesale figuresLand a hotel's Christmas orderThe freezer fails againKeep the fridge and freezer temperature logPut together a bakery's fire claim
An app to sort out the household billsAn app for a home‑loan brokerThe broker's app, with a WhatsApp assistantOnline ordering for a bakeryAn app for a piano teacher's studioAn app to run a laundry shopA piano practice app for a familyAn app for a gym and its membersA train‑ticket catcher for a commuterA smart home run from an iPadA fault finder for cooling unitsAssembly manuals for flat‑pack furnitureA journey planner for a city's transport
A customer who only wants a return labelDiagnose 80 customers at a repair deskAnswer an overdue complaintFind a customer's missing paymentsSettle a warranty claim from photosFind holiday rentals in a chatList every rental that fits
Run a delivery fleet for a monthRun a tea factory's cash for 13 weeksRun the spare‑parts stores at two sitesBuy everything for a new packing lineBuy spare parts as suppliers withdraw themPlan a coffee roaster's materialsHelp a tea factory's crew in its group chatDesign a tea factory's solar and batteryRun a coffee roaster's order desk at Christmas
Coordinate a hospital ward for three daysBuild a radiation‑dose records programBuild an emergency department's patient summary
Handle a server room's cooling failureFind why a server hall's temperature swingsMove an 800 kg air handler onto its plinth
Digitise a plant's scanned drawingsDigitise a plant's hand‑sketched drawingsBring a plant's drawings up to dateLeave a watchdog in charge of a plant
Furnish a room from six photosReplace a family's flooded furnitureGet a house ready for a family of sixPlan a homeowner's bedroom lightingFix a wet condo floor overnightQuote rooftop solar for a row house
Move a beam with two robot armsTurn and place three boardsPlace two boardsScrew two overlapping boards togetherBuild a three‑board cornerBuild a three‑board bridgeBuild a five‑board gridBuild a nine‑board gridBuild a slatted benchBuild a dog house's frame and roofBat ten pitches with two robot arms
Rescue a cancelled trip
The Calders' flight to a family wedding near Lisbon is cancelled, and the AI, at their travel firm, has to get all four there by Friday, spending up to $1,500 without asking.
81 out of 100Best score, from Anthropic's Opus 5.5
- Got all four to the wedding every time.
- Lost the airline's free seats while it checked, which left the family $680 out of pocket.
What AIs do well, and where they fall short
What they do well
Given explicit rules, the strongest can get a job exactly right.
At a repair desk with a policy and manuals, three of the five AIs we tested got all 80 customers right.
See the repair desk →The strongest can run a long job on their own.
Opus 5.5 got everything right on four of the eight jobs that run for weeks, among them a delivery fleet for a month.
See the delivery fleet →Where they fall short
They rarely pick up how you do things from old emails and chats.
How the company likes things done was only in its emails, chats and files: the AI got everything right in 2 of 130 tests at the bakery chain and brewery. With the rules written into the request, 50 of 130 did.
See the office jobs →They often don't ask a question unless told they can.
Three of a new client's charges needed her answer. In all 24 tests, the AI finished without asking her, but told they could, the three strongest asked every time.
See the new client's bills →They share more than they should.
On an insurance claim, 13 of 14 AIs passed on a staff member's worry that she'd caused the breakdown, though the evidence had ruled it out.
See the insurance claim →Which AIs did best?
Anthropic's Opus 5.5 and OpenAI's GPT‑6 Astra and GPT‑6.1 Sol came out on top, and they were close.
79
78
76
70
35
Opus 5.5Anthropic
GPT‑6 AstraOpenAI
GPT‑6.1 SolOpenAI
Sonnet 5.5Anthropic
GPT‑6 LunaOpenAI
Large
Mid‑sized
Small
Where each did best
- Opus 5.5 did best on the long jobs.
- GPT‑6 Astra did best in the office.
- GPT‑6.1 Sol got everything right on six jobs, among them the delivery fleet.
Bigger usually does better, but not always: GPT‑6.1 Sol, a mid‑sized AI, kept up with the two strongest large ones.
Average score out of 100, on the 62 jobs we tested all five on. We've tested one Google AI so far, on one job, and haven't yet tested Microsoft's.
Who we are
Machine learning researchers with a combined 60+ years of experience.
Founding advisor
Dr. Stefano Ermon
- Stanford professor
- Diffusion models inventor
- Inception Labs founder
Team affiliations
