How we test AI agents

We give an AI agent an everyday job in a world we made up, let it work on its own, and check the result in detail.

A request in plain words: Each job starts with a request written the way a person would write it, often short and incomplete, like a real message to an assistant.
A world we made up: The AI gets what a person doing that job would have. That can mean files, emails, chats and spreadsheets, apps and websites we built to work like real ones, and people to contact. We made up the businesses, families and people, as realistic as we could make them, so every AI meets the same world. On the two holiday‑rental jobs, the listings and guest reviews are real, with names removed.
The people in these worlds: They aren't real people, and they only know what we wrote for them. Usually code decides which fact a relative, customer or colleague may give, and only when asked the right question. On many jobs a separate AI, run apart from the one being tested, then puts that fact in the person's words, and each reply is checked for figures or names it made up. On other jobs, code writes the replies.
The AI works on its own: Nobody guides it. Some jobs take minutes. The long ones run for simulated weeks, and the AI is woken each day or each week and remembers only the notes it left itself.
Checked in detail: Almost every job has its own list of checks, from 14 to 919. A script decides most of them, going by what the AI actually did, like what it booked, paid, sent and wrote. Where a script can't decide, AI judges read the work, usually two from different makers, and when they disagree, a written ruling settles it. Three office jobs are scored mostly on money instead, by how close the AI's plan came to the best one.
Only everything right counts as done: We count a job as done only when an AI meets every check. On the job pages we describe what each AI got right and where it fell short.
Scores: For comparisons, a score out of 100 is the share of a job's checks an AI got right, weighted so the important ones count more, or, on the three money‑scored jobs, how close its plan came to the best. Where an AI had several tries, its score is the average. Some jobs have make‑or‑break checks, like every dinner guest having a seat: miss one and most of the score goes.
Tries and versions: Most jobs had one try per AI; ten had two or three, to show how steady the AIs are. On some jobs we also ran two other versions of the request, one with a short hint and one with every rule the job depends on written in, to show what changes when you tell the AI more. On two office jobs, some tries ran on an earlier version of the files: 24 of 39 on the broken ice cream freezer, scored against the best plan their records allowed, and two of GPT‑6 Astra's three on the temperature log, scored on the same checks as every other try.
Cost: Costs are what a try would cost a business at published per‑use prices (the makers' own, or a cloud provider's), not what you pay for an app like ChatGPT. They weren't recorded for every try, and a job page gives them only where most AIs have one. Where an AI has no published price, its cost is an estimate, and the page says so.
Who we tested, and when: 23 AIs from eight makers, between September and November 2026. We didn't test every AI on every job. We sort most of them into large, mid‑sized and small, as a rough guide. We've tested one Google AI so far and none of Microsoft's, and we don't yet have people's scores on the same jobs to compare.