AsapiYour AI operations manager
We built eight jobs that run for simulated weeks, at a parcel company, a tea factory and a coffee roaster, all made up. The AI dispatches the parcel company's fleet, runs the tea factory's cash desk and helps its crew in their group chat. At the coffee roaster it buys for a new packing line, keeps parts, coffee and packaging coming, and runs the web shop's Christmas orders. Each time we wake it, it remembers only its own notes.
8 jobs · 8 AIs tested. Opus 5.5 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.
The eight jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
Run a delivery fleet for a month (2 AIs tested)
The AI plans a delivery company's trucks and gives every customer a delivery window, day after day for a simulated month.
What the best AI did
- Ran the fleet for all 28 days, kept trucks off potholed roads, and none broke down.
How each AI scored
Run a tea factory's cash for 13 weeks (5 AIs tested)
Each Monday, the AI decides what to sell to whom and when to ask the owner for money, without running out of cash.
What the best AI did
- Never ran out of cash in any of the 16 different quarters we ran, and asked the owner only when needed.
How each AI scored
Run the spare‑parts stores at two sites (6 AIs tested)
The AI keeps two sites' spare‑parts stores stocked, so technicians find what each repair needs.
What the best AI did
- Ran the stores for eight weeks, learning from the logs which parts each repair takes.
How each AI scored
Buy everything for a new packing line (6 AIs tested)
The AI buys every part the company has to supply for a new packing line, in time for each stage of the installation.
What the best AIs did
- Had every item on site in time without buying what others supply or the stores already hold, in all three tries.
How each AI scored
Buy spare parts as suppliers withdraw them (5 AIs tested)
The AI keeps spare parts on the shelf at two sites as suppliers stop making six of them.
What the best AIs did
- Chose the right replacement for every withdrawn part, and no repair waited more than a shift.
How each AI scored
Plan a coffee roaster's materials (5 AIs tested)
The AI buys coffee and packaging in time for a roaster's weekly plan, through a port closure and the Christmas rush.
What the best AI did
- Kept every customer's lines on time and in full through the four‑week Christmas peak, and had suppliers hold coffee for each sales tip that became a real order.
- Ran short of two coffees in one November week, when a port closure held up shipments longer than it planned for, and two customers got short deliveries.
How each AI scored
Help a tea factory's crew in its group chat (6 AIs tested)
In the crew's group chat, the AI speaks up when a machine setting is wrong for the day's tea leaf, and stays quiet when the crew has it right.
What the best AI did
- Learned much of how the crew works, and mostly held back when the crew was right.
- Caught 27 of the 47 real problems, and missed the rest.
How each AI scored
Run a coffee roaster's order desk at Christmas (5 AIs tested)
The AI runs a coffee roaster's web‑shop order desk through the Christmas peak, from the warehouse paperwork to every customer's case, waking each working day with only the notes it left itself.
What the best AI did
- Got orders out on the day the shop promises right through the Christmas peak, and claimed for most of the carrier's late parcels.
- Answered dozens of customers' cases differently from the shop's policy, in three of the four stretches.
How each AI scored
We gave each AI three tries on the packing line, and Opus 5.5 and Sonnet 5.5 two on the coffee roaster's materials planning, to see how steady they were. There, an AI only counts as getting a job right if every try did, and its score is the average.
Who did best
Anthropic's Opus 5.5, the only AI we tested on all eight, got everything right on four: the delivery fleet, the tea factory's cash, the spare‑parts stores and the packing line, where GPT‑6 Astra, GPT‑6.1 Sol and Fable 5.1 did too. Astra and Sol were the only AIs to get everything right buying spare parts as suppliers withdrew them, and Sol did best on the coffee roaster's materials. Sonnet 5.5, tested on seven, got none right.
Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on more than half of the eight jobs.
What the eight jobs showed
What they do well
They can keep a business running for weeks from their own notes.
The strongest tries wrote themselves a routine on day one, and Opus 5.5's morning routine ran the delivery fleet for 28 days without a missed day. At the coffee roaster, it got the web shop's orders out on the day the shop promises right through the Christmas peak.
They can buy in time for what's planned.
On the packing line, four AIs had every item on site in time, without buying what others supply or the stores already hold, in all three of their tries.
Where they fall short
They act too late on routine things.
Planning the coffee roaster's materials, Sonnet 5.5 didn't re‑order cartons it had noted it needed until it was too late, and the season ended in week 8. At the tea factory's cash desk, asking the owner for money too late sank the factory for two AIs.
They lose track of stock.
At the spare‑parts stores, two AIs reported their stock within the limit while it was well over. Through Christmas, Opus 5.5 didn't take coffees off the web shop before they sold out.
How these jobs were set up and scored
- The businesses
- Each job runs in a simulated business, from a city's roads and trucks to a factory's books and bank and a web shop's orders. The same scripted events happen on the same days for every AI.
- No memory
- Each session starts from nothing, and the AI keeps only the notes and tools it leaves in its own folder. In the tea factory's chat, its notes were limited to a few pages, written once.
- Woken by events
- Sessions open each morning, each Monday, at set times or when mail arrives, from 9 sessions on the spare‑parts stores to as many as 208 at the cash desk.
- Checking
- It's all done by script, from each simulation's own record. There are no AI judges on these jobs, and each job has one request, with no hint or rules written in.
- Scores
- A score is out of 100, for how much of a job an AI got right, averaged over tries, and each job's page says how it works. On some jobs an AI that did nothing would still score: 50 in the tea factory's chat, 39 buying spare parts and 18 running the stores.
- Fewer AIs
- We tested two to six AIs on each job, and Opus 5.5 on all eight.
Every AI's score on every job
| AI | Run a delivery fleet for a month | Run a tea factory's cash for 13 weeks | Run the spare‑parts stores at two sites | Buy everything for a new packing line | Buy spare parts as suppliers withdraw them | Plan a coffee roaster's materials | Help a tea factory's crew in its group chat | Run a coffee roaster's order desk at Christmas |
|---|---|---|---|---|---|---|---|---|
| Opus 5.5Anthropic | 100 | 100 | 100 | 100 | 97 | 91 | 63 | 94 |
| GPT‑6 AstraOpenAI | – | 81 | 75 | 100 | 100 | 94 | 54 | 86 |
| GPT‑6 LunaOpenAI | – | 12 | 95 | 94 | 90 | 26 | 40 | 61 |
| GPT‑6.1 SolOpenAI | – | 88 | 70 | 100 | 100 | 97 | 41 | 90 |
| Sonnet 5.5Anthropic | 25 | 62 | 98 | 97 | 98 | 50 | – | 92 |
| Fable 5.1Anthropic | – | – | 98 | 100 | – | – | – | – |
| DeepSeek V4.1 Flash openDeepSeek | – | – | – | – | – | – | 44 | – |
| Opus 5Anthropic | – | – | – | – | – | – | 67 | – |
These are the scores out of 100, averaged where an AI had two or three tries. Darker means higher. A dash means the AI has no result on that job. Open marks an AI anyone can download.