Asapi / Catalogue
80 environments, grouped by area
One row per environment: how it works, how it's graded, and how the models we tested did with only the request, as a user would give it. Grouped by area, as on The AI Agent Trials. Filter by capability.
| Environment | Time | People | Grader | Checks | Attempts | Median | Best | Best open | Solved | Told the rules | Minutes | Tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Personal assistant | ||||||||||||
| Family dinner booking with simulated relativesHousehold | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 93 | 12 × 3 | 10% | 93% | 12% | 0 of 36 | 97% | 18 | 2.7M |
| Cancelled‑flight family rebooking against a ticking clockTravel conciergeCaveat: A single read‑only lookup scores about 20% | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 61–63 | 8 × 3 | 70% | 85% | – | 0 of 24 | 99% | 10 | 10.5M |
| Household bill review: card disputes, claims, trackerHousehold bills | Single session | None | Script + LLM judges | 191–262 | 8 × 3 | 86% | 94% | – | 0 of 24 | – | 12 | 17.8M |
| Onboarding bill review with a simulated clientHousehold bills | Single session | LLM‑voiced | Script + LLM judges | 295–320 | 8 × 3 | 78% | 84% | – | 0 of 24 | 95% | 10 | 10.9M |
| Medical bills review across two health plansHousehold bills | Single session | LLM‑voiced | Script + LLM judges | 287–350 | 8 × 3 | 74% | 88% | – | 0 of 24 | 99% | 16 | 14.3M |
| Household solar purchase plan with simulated contactsHousehold | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 82 | 10 × 1 | 24% | 54% | 48% | 0 of 10 | 96% | 12 | 1.6M |
| Household savings plan from records and familyHousehold | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 51 | 8 × 1 | 50% | 85% | – | 0 of 8 | 98% | 8 | 1.5M |
| Course registration against rules found in portalUniversity portalCaveat: An empty registration meets 11 of 14 checks | Single session | None | Script | 14 | 16 × 1 | 71% | 100% | – | 2 of 16 | – | 6 | 636k |
| Course registration optimised for earliest graduationUniversity portalCaveat: An empty registration meets 11 of 14 checks | Single session | None | Script | 14 | 15 × 1 | 79% | 93% | – | 0 of 15 | – | 10 | 1.4M |
| Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantCaveat: A legal early flight chosen without asking about the papers scores about 90% | Single session | LLM‑voiced | Script + LLM judges | 34 | 8 × 3 | 61% | 96% | – | 0 of 24 | 99% | 9 | 8.7M |
| Office assistant | ||||||||||||
| Packaging proof sign‑off for an absent ownerSmall business | Single session | None | Script + LLM judges | 15 | 13 × 1 | 2% | 84% | 2% | 0 of 13 | 100% | 7 | 1.7M |
| 12‑month equipment plan with simulated shop managerSmall business | Single session | LLM‑voiced | Cost model | Cost model | 13 × 1 | 0% | 57% | 0% | 0 of 13 | 86% | 3 | 82k |
| Monthly waste report in house formatSmall businessCaveat: A bare table with none of the asked‑for notes scores 49% | Single session | None | Script + LLM judges | 43 | 13 × 1 | 64% | 89% | 76% | 0 of 13 | 100% | 7 | 1.4M |
| Brewery input‑price report in house formatSmall businessCaveat: A bare table with none of the asked‑for notes scores 64% | Single session | None | Script + LLM judges | 51 | 12 × 1 | 73% | 90% | 71% | 0 of 12 | 98% | 8 | 1.0M |
| Contractor invoice audit against shop recordsSmall business | Single session | None | Script + LLM judges | 107 | 13 × 1 | 44% | 61% | 44% | 0 of 13 | 95% | 15 | 2.7M |
| Bakery's freezer‑loss insurance claim packSmall business | Single session | None | Script + LLM judges | 60 | 14 × 1 | 55% | 83% | 83% | 0 of 14 | 100% | 11 | 1.5M |
| Bakery wholesale pack with downstream figure checksSmall business | Single session | None | Script + LLM judges | 23 | 13 × 1 | 48% | 90% | 52% | 0 of 13 | 100% | 13 | 1.7M |
| Seasonal bulk order run with simulated staffSmall business | In‑world clock | LLM‑voiced + scripted | Cost model + LLM judges | 23 | 8 × 1 | 46% | 75% | – | 0 of 8 | 95% | 17 | 1.9M |
| Three‑year repair‑or‑replace plan with simulated managerSmall business | Single session | LLM‑voiced | Cost model | Cost model | 12 × 1 | 0% | 93% | 8% | 0 of 12 | 97% | 8 | 181k |
| Temperature compliance register from messy logger dataSmall business | Single session | None | Script | 103 | 12 × 1 | 9% | 40% | 6% | 0 of 12 | 77% | 46 | 12.6M |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 29 | 7 × 1 | 81% | 99% | – | 0 of 7 | 99% | 11 | 3.0M |
| App builder | ||||||||||||
| Bill‑audit and claims app from household filesHousehold | Single session | None | Tester model + judges | 108 | 8 × 1 | 57% | 61% | – | 0 of 8 | 76% | 14 | 3.2M |
| Refinance‑broker case app from a non‑coder's laptopSmall business | Single session | None | Tester model + judges | 51 | 8 × 1 | 75% | 90% | – | 0 of 8 | 98% | 19 | 2.8M |
| Broker app with autonomous lead‑messaging assistantSmall business | Single session | None | Tester model + judges | 106 | 8 × 1 | 58% | 68% | – | 0 of 8 | 91% | 19 | 2.6M |
| Multi‑shop bakery ordering app from business recordsSmall business | Single session | None | Tester model + judges | 97 | 8 × 1 | 60% | 75% | – | 0 of 8 | 99% | 7 | 1.5M |
| Music‑teaching studio app: lessons, billing, mediaSmall business | Single session | None | Tester model + judges | 155–179 | 7 × 1 | 76% | 81% | – | 0 of 7 | 94% | 20 | 9.0M |
| Laundry‑shop operations app for owner, hosts, workersSmall business | Single session | None | Tester model + judges | 161 | 7 × 1 | 65% | 77% | – | 0 of 7 | 81% | 39 | 4.7M |
| Listening piano‑practice app: following, transcription, feedbackHousehold | Single session | None | Tester model + judges | 64 | 7 × 1 | 53% | 58% | – | 0 of 7 | 62% | 41 | 5.3M |
| Gym operations app with three owner‑feedback repair roundsSmall business | Single session | LLM‑voiced | Tester model + judges | 113 | 7 × 1 | 82% | 86% | – | 0 of 7 | 86% | 35 | 9.2M |
| Unattended train‑ticket buyer for a rate‑limited booking siteHousehold | Single session | None | Tester model + judges | 26 | 8 × 1 | 83% | 100% | – | 1 of 8 | 100% | 12 | 2.0M |
| Smart‑home retrofit: purchase plan, wiring and local control softwareHousehold | Single session | None | Tester model + judges | 121 | 7 × 1 | 76% | 92% | – | 0 of 7 | 96% | 32 | 11.3M |
| Offline fault‑finding tablet app for data‑centre cooling unitsFacilities | Single session | None | Tester model + judges | 79 | 7 × 1 | 64% | 87% | – | 0 of 7 | 83% | 27 | 5.8M |
| Flat‑pack assembly manuals with 3D views from unlabelled photosHousehold | Single session | None | Tester model + judges | 173 | 7 × 1 | 53% | 63% | – | 0 of 7 | 64% | 33 | 3.7M |
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Multi‑session | Scripted | Script | 81 | 5 × 1 | 84% | 98% | – | 0 of 5 | – | 115 | 23.8M |
| Customer service | ||||||||||||
| Returns‑label request hiding unraised account problemsCustomer service | Single session | None | Script + LLM judges | 62 | 8 × 3 | 94% | 98% | – | 0 of 24 | 100% | 4 | 1.6M |
| Appliance repair triage with budgeted customer questionsCustomer service | 80 separate sessions | Scripted | Script | 80 | 5 × 1 | 100% | 100% | – | 3 of 5 | – | 94 | 37.7M |
| Retail escalation reply and case recordsCustomer service | Single session | None | Script + LLM judges | 114–115 | 15 × 1 | 73% | 98% | 30% | 0 of 15 | 100% | 6 | 2.8M |
| Missing‑payments reply from error‑laden retail recordsCustomer service | Single session | None | Script + LLM judges | 61 | 8 × 3 | 78% | 99% | – | 0 of 24 | 100% | 5 | 2.3M |
| Furniture warranty claim from a simulated customer's photosCustomer service | Single session | LLM‑voiced | Script + LLM judges | 56 | 8 × 3 | 82% | 95% | – | 0 of 24 | 99% | 6 | 4.2M |
| Multi‑turn holiday‑rental search chat with saved profilesCustomer service | Single session | Scripted | Script | 25 | 6 × 1 | 34% | 88% | – | 0 of 6 | – | 18 | 13.1M |
| Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer service | Single session | None | Script | 20 | 6 × 1 | 88% | 100% | – | 1 of 6 | – | 13 | 3.8M |
| Operations manager | ||||||||||||
| Month‑long parcel fleet dispatch, notes‑only memorySmall business | 28 simulated days | Scripted | Simulator | 29 | 4 × 1 | 62% | 100% | – | 2 of 4 | – | 158 | 52.7M |
| Tea‑factory cash desk over thirteen weekly sessionsSmall business | 13 weekly sessions | Scripted | Simulator | 64 | 5 × 1 | 81% | 100% | – | 1 of 5 | – | 559 | 129.5M |
| Two‑site spares pick lists and stock plansManufacturing | 8 weekly sessions | None | Simulator | 35 | 6 × 1 | 90% | 100% | – | 1 of 6 | – | 22 | 4.5M |
| Packing‑line project purchasing over twelve mail‑driven weeksManufacturing | 12 simulated weeks | Scripted | Simulator | 14 | 6 × 3 | 100% | 100% | – | 13 of 18 | – | 37 | 9.3M |
| Two‑site spares buying through supplier part withdrawalsManufacturing | 12 simulated weeks | Scripted | Simulator | 62 | 5 × 1 | 97% | 100% | – | 2 of 5 | – | 61 | 15.4M |
| Coffee roaster materials planning across two seasonsManufacturingCaveat: Claude Code's 4 attempts ran outside a container, beside the simulator and grader; transcripts were checked afterwards | Two 16‑week windows | Scripted | Simulator | 112–166 | 5 × 1–2 | 90% | 98% | – | 0 of 7 | – | 113 | 43.9M |
| Tea‑factory ops chat: speak or stay silentSmall business | Short check‑in sessions | None | Script | 71 | 6 × 1 | 11% | 53% | 2% | 0 of 6 | – | 172 | 14.3M |
| Tea factory rooftop solar and battery design with its business caseEnergy engineering | Single session | None | Simulator | 18 | 6 × 1 | 66% | 92% | – | 0 of 6 | – | 21 | 5.5M |
| Coffee web‑shop order desk through a four‑act Christmas peakSmall business | Multi‑session | Scripted | Simulator | 207–213 | 5 × 1 | 90% | 94% | – | 0 of 5 | – | 306 | 158.9M |
| Hospital assistant | ||||||||||||
| Three‑day hospital ward care coordination, notes‑only memoryHospital | 72 simulated hours | Scripted | Simulator | 290–415 | 6 × 1 | 95% | 100% | 96% | 1 of 6 | – | 135 | 54.1M |
| Radiation‑dose registry program from a long specHospital software | Single session | None | Hidden tests | 466 | 12 × 1 | 100% | 100% | 100% | 8 of 12 | – | 14 | 2.7M |
| Emergency patient‑history compiler from a long specHealth records software | Single session | None | Hidden tests | 919 | 6 × 1 | 99% | 100% | – | 2 of 6 | – | 36 | 15.6M |
| Facilities engineer | ||||||||||||
| Server‑room cooling outage in a 3D building with model‑played staffFacilities | Single session | LLM‑voiced | Script + LLM judges | 31 | 7 × 1 | 5% | 33% | – | 0 of 7 | 92% | 6 | 1.6M |
| Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilities | Single session | LLM‑voiced | Script + LLM judges | 20 | 7 × 1 | 35% | 50% | – | 0 of 7 | 95% | 6 | 1.4M |
| 800 kg air handler move proved by trial in a 3D physics modelFacilities | Single session | Scripted | Script | 18 | 7 × 1 | 33% | 44% | – | 0 of 7 | 100% | 29 | 12.7M |
| Engineering assistant | ||||||||||||
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | Single session | Scripted | Script | 3,836 | 5 × 1 | 99% | 100% | – | 1 of 5 | – | 17 | 1.3M |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | Single session | Scripted | Script | 3,836 | 5 × 1 | 99% | 99% | – | 0 of 5 | – | 18 | 2.6M |
| P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering records | Single session | Scripted | Script | 65 | 6 × 1 | 97% | 100% | – | 1 of 6 | – | 11 | 4.2M |
| Watchdog program commissioned for a running solvent recovery unitProcess controlCaveat: An idle agent earns 12%, though the grader fails it | In‑world clock | Scripted | Simulator | 41 | 6 × 1 | 79% | 93% | – | 0 of 6 | – | 28 | 5.4M |
| DIY assistant | ||||||||||||
| Room furnishing from photos: measure, match look‑alikes, order and planFurnishing service | Single session | None | Script | 28 | 8 × 3 | 63% | 80% | – | 0 of 24 | 99% | 16 | 10.3M |
| Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacement | Single session | None | Script | 25 | 8 × 3 | 75% | 85% | – | 0 of 24 | 100% | 13 | 15.8M |
| Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove management | Single session | None | Script | 45 | 8 × 3 | 83% | 92% | – | 0 of 24 | 100% | 19 | 12.8M |
| DIY smart‑lighting plan from notes, photos and one shop's catalogueHousehold | Single session | LLM‑voiced | Script + LLM judges | 26 | 8 × 1 | 89% | 94% | – | 0 of 8 | 100% | 7 | 917k |
| Night repair of a leaking condo floor on an in‑world clockHousehold | In‑world clock | None | Script + LLM judges | 13 | 7 × 3 | 41% | 100% | – | 1 of 21 | 100% | 2 | 254k |
| Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solar | Single session | None | Simulator | 15 | 6 × 1 | 71% | 99% | – | 0 of 6 | – | 37 | 4.3M |
| Robot operator | ||||||||||||
| Flat‑beam transfer by a bimanual robot with friction‑only gripsRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 2 | – |
| Three angled boards turned to flat outlines by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 8 | – |
| Two boards placed together by a bimanual robot, one per handRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 12 | – |
| Screwed two‑board lap joint by a bimanual robot with a hand‑held drillRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Screwed three‑board corner frame by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Screwed three‑board bridge by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Screwed five‑board grid from angled stock by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Screwed nine‑board grid with long rails by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Upright slatted bench from 11 boards by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Dog‑house frame and roof from 19 boards by a bimanual robotRobot manipulation | Single session | None | Script | 1 | 1 × 1 | 0% | 0% | – | 0 of 1 | – | 20 | – |
| Baseball batting by a bimanual robot from batter‑eye imagesVisual robot manipulation | Single session | None | Script | Points | 4 × 1 | 0% | 0% | – | – | – | – | – |
| Race team | ||||||||||||
| Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineering | Single session | Scripted | Script | 4 | 5 × 1 | 4.6% slower | 2.0% slower | – | 0 of 5 | – | 19 | 20.0M |
| Four‑car race driven from cockpit images on a live physics clockMotor racing | In‑world clock | None | Script | 1 | 4 × 1 | 100% | 100% | – | 4 of 4 | – | 12 | – |
| Four‑car race on a live physics clock with lasting crash damageMotor racing | In‑world clock | None | Script | 1 | 4 × 1 | 100% | 100% | – | 3 of 4 | – | 18 | – |
- Setting
- Where the job happens. Several environments share one: eleven are at one bakery chain and five at one coffee roaster, each with a different job.
- Checks
- The requirements the grader checks, each earning part of the reward; a range where the count varies by attempt.
- Attempts
- Models × attempts per model, with only the request; a range where models made different numbers.
- Median, best
- The median and the best attempt's reward, from 0 to 100%; for the race car, how much slower the car lapped than the fastest achievable car. More attempts give the best more chances.
- Best open
- The best attempt by an open‑weight model, where one was tested.
- Solved
- Full solves: attempts that met every requirement; for the race car, the fastest achievable car itself.
- Told the rules
- The best attempt when the request also gave the rules or method the job depends on; in 11 environments also facts from the files or people, in five of them figures the grader checks. A dash: not tested.
- Minutes, tokens
- The median attempt's wall‑clock time, over every attempt, and its tokens, cached input included; for tokens, where each model made several attempts, the median of the models' means.
- Cost model
- The reward prices the agent's plan from a baseline (0) to the best of tens of thousands of candidates (1), times the share of key facts quoted. In the bulk order, profit is 70%, up to the best outcome its staff can carry out, and checks 30%, less penalties.
- Caveat
- A known weakness in an environment's grader or setup, under its name. Methods lists them all.
- Percentages
- Rounded to the nearest point, except that only a reward of 1 shows as 100%; the race car's to one decimal.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials