What 56 jobs taught us about AI agents

23 AIs, large and small, took on 56 jobs between them, in worlds we made up. Here's what they did well, where they fell short, and how much size mattered.

What they do well

Given explicit rules, the strongest can get a job exactly right.

Most of the jobs where an AI got everything right came with explicit rules or a manual.

We built an appliance repair company with a policy, a guide and manuals that spell out its rules, and put the AIs on its service desk, one at a time, with 80 customers each. Of the five we tested, Opus 5.5, GPT‑6 Astra and GPT‑6.1 Sol each got all 80 right, down to the parts and the price.
GPT‑6 Astra and GPT‑6.1 Sol each built a radiation‑dose records program for hospitals from a 2,448‑line manual, and both passed all 466 of our checks.
Opus 5.5 coordinated a made‑up hospital ward for three days and nights from its policy manual, and got everything right.

What they do well

The strongest can run a long job on their own.

We left AIs in charge for weeks at a time. Each time we woke them, they remembered only the notes they'd left themselves.

Opus 5.5, the only AI we tested on all eight long jobs, got everything right on four: a parcel company's delivery fleet, a tea factory's cash, a coffee roaster's spare‑parts stores and the buying for its new packing line.
On all 28 days of the fleet job, nearly every parcel arrived in its two‑hour window. Opus 5.5 kept its trucks off the roads drivers said were badly potholed and sent worn trucks for a service early, and none broke down.
GPT‑6 Astra and GPT‑6.1 Sol got everything right as the roaster's spare‑parts buyer for twelve weeks. No repair waited more than a shift for a part.

Where they fall short

They rarely pick up how you do things from old emails and chats.

We built a bakery chain, a brewery and a furnishing service and gave the AIs twelve office jobs there. On most of them, the AI got the staff's own emails, chats and files. How the business likes things done was in there, but scattered across those files.

GPT‑6 Astra, which came closest, missed something on ten of the twelve jobs.
The bakery chain's owner said once, in a staff chat, that one customer should stay out of every total in the monthly wholesale figures. Last month's figures still counted that customer, and so did 12 of 13 AIs.
At the bakery chain and brewery, when we wrote what each job depends on into the request, 50 of 130 tries got everything right. When we didn't, 2 of 130 did.

Where they fall short

They often don't ask a question unless told they can.

In several jobs, only one person knew something the AI needed. Whether it asked often came down to whether it had been told it could.

A bill‑checking firm we built took on a new client, and a few of her charges could only be explained by her. The AI could message her, but nothing in the request said so. We tested eight AIs, three times each, and not one try asked her. Then we added one sentence saying she could be messaged, and gave that to the three strongest AIs. They had five tries between them, and asked her every time.
A father planning his family's savings said he couldn't answer questions, though his wife and HR could. Six of eight AIs asked nobody. A one‑line hint got all eight asking.
We tested nine AIs on organising a big family's New Year dinner, and most messaged relatives. But one relative passed on a rumour that a family abroad wasn't coming. Seven of the nine never asked that family themselves, and six left them out.

Where they fall short

They share more than they should.

AIs writing on someone's behalf often passed on things the reader had no need to know.

On an insurance claim for a broken shop freezer, 13 of 14 AIs told the insurer that a member of staff was worried she had caused the breakdown, though the evidence had already ruled that out.
All 21 letters to a husband's employer about a couple's medical bills described his medical care, where a claim number, date and amount would have done.
Three of the nine AIs organising the family dinner told the family about the cashback on the organiser's credit card, which his notes say he keeps for himself.

Where they fall short

They can say it's done when it isn't.

Several AIs reported success the work didn't bear out, and some went ahead with things they had doubts about.

We asked 13 AIs to fill in a printer's approval forms for a new cake box, in a bakery owner's name. Twelve of them approved a box that left the tallest cake almost no room under the lid. Opus 5.5 and one other worked out how little room there was and approved it anyway to keep the printing date. They mentioned the risk only in their chat replies, not on the forms.
Six of the nine AIs organising the family dinner told the organiser how many were coming and left out the family abroad. Only one said it might have missed someone.
One AI building the radiation‑dose records program reported its work complete and checked. Our tests failed it on 10 checks, all from one rule it had got wrong.
Another, asked to build an app from a detailed list of requirements, said it had checked the whole thing from start to finish. The app didn't start.

Where they fall short

They can build you an app, but it misses basics nobody asked for by name.

We invented eleven people who can't code, and they asked for twelve apps. Several AIs built each one. The best did much of what their owners asked, but only one of 89 tries built everything its owner needed: Opus 5.5's train‑ticket catcher.

A woman sorting out her bills had 12 wrong charges in her 333 files. Eight AIs each built her an app, and three of those apps got 10 of the 12 right, with the right amount and the right place to send each claim. But none of the eight had a sign‑in.
All eight apps built for a home‑loan broker wrote his real customers' details into the program itself, so they go wherever the program goes. When we ran the job again with a written how‑to, one line of it asked for made‑up examples instead, and all eight kept his customers out.

How much size matters

Bigger usually does better, but not always.

AI makers offer models in different sizes. We sort most of the ones we tested into large, mid‑sized and small, as a rough guide. On the 52 jobs where we tested AIs of all three sizes, a large AI got everything right on 11, a mid‑sized one on 6 and a small one on 1.

GPT‑6.1 Sol, a mid‑sized AI, kept up with the two strongest large ones, Opus 5.5 and GPT‑6 Astra.
GPT‑6 Luna, the small AI we tested on the most jobs, got everything right on none of them.
We tested four small AIs on the radiation‑dose records program, and two of them, DeepSeek V4.1 Flash and Qwen3.8‑27B, got everything right, while one large AI didn't.
DeepSeek V4.1 Flash also tied for closest on the freezer's insurance claim. On the hospital ward, which both failed, it got a larger share of its checks right than GPT‑6 Astra, a large AI.

How we test →