AsapiThe AI Agent Trials
Asapi / Catalogue
Catalogue

80 environments, grouped by area

One row per environment: how it works, how it's graded, and how the models we tested did with only the request, as a user would give it. Grouped by area, as on The AI Agent Trials. Filter by capability.

EnvironmentTimePeople GraderChecksAttemptsMedianBestBest openSolvedTold the rulesMinutesTokens
Personal assistant
Family dinner booking with simulated relativesHouseholdIn‑world clockLLM‑voiced + scriptedScript + LLM judges9312 × 310%93%12%0 of 3697%182.7M
Cancelled‑flight family rebooking against a ticking clockTravel conciergeCaveat: A single read‑only lookup scores about 20%In‑world clockLLM‑voiced + scriptedScript + LLM judges61–638 × 370%85%–0 of 2499%1010.5M
Household bill review: card disputes, claims, trackerHousehold billsSingle sessionNoneScript + LLM judges191–2628 × 386%94%–0 of 24–1217.8M
Onboarding bill review with a simulated clientHousehold billsSingle sessionLLM‑voicedScript + LLM judges295–3208 × 378%84%–0 of 2495%1010.9M
Medical bills review across two health plansHousehold billsSingle sessionLLM‑voicedScript + LLM judges287–3508 × 374%88%–0 of 2499%1614.3M
Household solar purchase plan with simulated contactsHouseholdIn‑world clockLLM‑voiced + scriptedScript + LLM judges8210 × 124%54%48%0 of 1096%121.6M
Household savings plan from records and familyHouseholdIn‑world clockLLM‑voiced + scriptedScript + LLM judges518 × 150%85%–0 of 898%81.5M
Course registration against rules found in portalUniversity portalCaveat: An empty registration meets 11 of 14 checksSingle sessionNoneScript1416 × 171%100%–2 of 16–6636k
Course registration optimised for earliest graduationUniversity portalCaveat: An empty registration meets 11 of 14 checksSingle sessionNoneScript1415 × 179%93%–0 of 15–101.4M
Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantCaveat: A legal early flight chosen without asking about the papers scores about 90%Single sessionLLM‑voicedScript + LLM judges348 × 361%96%–0 of 2499%98.7M
Office assistant
Packaging proof sign‑off for an absent ownerSmall businessSingle sessionNoneScript + LLM judges1513 × 12%84%2%0 of 13100%71.7M
12‑month equipment plan with simulated shop managerSmall businessSingle sessionLLM‑voicedCost modelCost model13 × 10%57%0%0 of 1386%382k
Monthly waste report in house formatSmall businessCaveat: A bare table with none of the asked‑for notes scores 49%Single sessionNoneScript + LLM judges4313 × 164%89%76%0 of 13100%71.4M
Brewery input‑price report in house formatSmall businessCaveat: A bare table with none of the asked‑for notes scores 64%Single sessionNoneScript + LLM judges5112 × 173%90%71%0 of 1298%81.0M
Contractor invoice audit against shop recordsSmall businessSingle sessionNoneScript + LLM judges10713 × 144%61%44%0 of 1395%152.7M
Bakery's freezer‑loss insurance claim packSmall businessSingle sessionNoneScript + LLM judges6014 × 155%83%83%0 of 14100%111.5M
Bakery wholesale pack with downstream figure checksSmall businessSingle sessionNoneScript + LLM judges2313 × 148%90%52%0 of 13100%131.7M
Seasonal bulk order run with simulated staffSmall businessIn‑world clockLLM‑voiced + scriptedCost model + LLM judges238 × 146%75%–0 of 895%171.9M
Three‑year repair‑or‑replace plan with simulated managerSmall businessSingle sessionLLM‑voicedCost modelCost model12 × 10%93%8%0 of 1297%8181k
Temperature compliance register from messy logger dataSmall businessSingle sessionNoneScript10312 × 19%40%6%0 of 1277%4612.6M
Kitchen‑fire contents claim from stale counts, invoices and photosSmall businessIn‑world clockLLM‑voiced + scriptedScript + LLM judges297 × 181%99%–0 of 799%113.0M
App builder
Bill‑audit and claims app from household filesHouseholdSingle sessionNoneTester model + judges1088 × 157%61%–0 of 876%143.2M
Refinance‑broker case app from a non‑coder's laptopSmall businessSingle sessionNoneTester model + judges518 × 175%90%–0 of 898%192.8M
Broker app with autonomous lead‑messaging assistantSmall businessSingle sessionNoneTester model + judges1068 × 158%68%–0 of 891%192.6M
Multi‑shop bakery ordering app from business recordsSmall businessSingle sessionNoneTester model + judges978 × 160%75%–0 of 899%71.5M
Music‑teaching studio app: lessons, billing, mediaSmall businessSingle sessionNoneTester model + judges155–1797 × 176%81%–0 of 794%209.0M
Laundry‑shop operations app for owner, hosts, workersSmall businessSingle sessionNoneTester model + judges1617 × 165%77%–0 of 781%394.7M
Listening piano‑practice app: following, transcription, feedbackHouseholdSingle sessionNoneTester model + judges647 × 153%58%–0 of 762%415.3M
Gym operations app with three owner‑feedback repair roundsSmall businessSingle sessionLLM‑voicedTester model + judges1137 × 182%86%–0 of 786%359.2M
Unattended train‑ticket buyer for a rate‑limited booking siteHouseholdSingle sessionNoneTester model + judges268 × 183%100%–1 of 8100%122.0M
Smart‑home retrofit: purchase plan, wiring and local control softwareHouseholdSingle sessionNoneTester model + judges1217 × 176%92%–0 of 796%3211.3M
Offline fault‑finding tablet app for data‑centre cooling unitsFacilitiesSingle sessionNoneTester model + judges797 × 164%87%–0 of 783%275.8M
Flat‑pack assembly manuals with 3D views from unlabelled photosHouseholdSingle sessionNoneTester model + judges1737 × 153%63%–0 of 764%333.7M
Transit journey planner and phone telemetry over five field weeksSoftware for a transit operatorMulti‑sessionScriptedScript815 × 184%98%–0 of 5–11523.8M
Customer service
Returns‑label request hiding unraised account problemsCustomer serviceSingle sessionNoneScript + LLM judges628 × 394%98%–0 of 24100%41.6M
Appliance repair triage with budgeted customer questionsCustomer service80 separate sessionsScriptedScript805 × 1100%100%–3 of 5–9437.7M
Retail escalation reply and case recordsCustomer serviceSingle sessionNoneScript + LLM judges114–11515 × 173%98%30%0 of 15100%62.8M
Missing‑payments reply from error‑laden retail recordsCustomer serviceSingle sessionNoneScript + LLM judges618 × 378%99%–0 of 24100%52.3M
Furniture warranty claim from a simulated customer's photosCustomer serviceSingle sessionLLM‑voicedScript + LLM judges568 × 382%95%–0 of 2499%64.2M
Multi‑turn holiday‑rental search chat with saved profilesCustomer serviceSingle sessionScriptedScript256 × 134%88%–0 of 6–1813.1M
Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer serviceSingle sessionNoneScript206 × 188%100%–1 of 6–133.8M
Operations manager
Month‑long parcel fleet dispatch, notes‑only memorySmall business28 simulated daysScriptedSimulator294 × 162%100%–2 of 4–15852.7M
Tea‑factory cash desk over thirteen weekly sessionsSmall business13 weekly sessionsScriptedSimulator645 × 181%100%–1 of 5–559129.5M
Two‑site spares pick lists and stock plansManufacturing8 weekly sessionsNoneSimulator356 × 190%100%–1 of 6–224.5M
Packing‑line project purchasing over twelve mail‑driven weeksManufacturing12 simulated weeksScriptedSimulator146 × 3100%100%–13 of 18–379.3M
Two‑site spares buying through supplier part withdrawalsManufacturing12 simulated weeksScriptedSimulator625 × 197%100%–2 of 5–6115.4M
Coffee roaster materials planning across two seasonsManufacturingCaveat: Claude Code's 4 attempts ran outside a container, beside the simulator and grader; transcripts were checked afterwardsTwo 16‑week windowsScriptedSimulator112–1665 × 1–290%98%–0 of 7–11343.9M
Tea‑factory ops chat: speak or stay silentSmall businessShort check‑in sessionsNoneScript716 × 111%53%2%0 of 6–17214.3M
Tea factory rooftop solar and battery design with its business caseEnergy engineeringSingle sessionNoneSimulator186 × 166%92%–0 of 6–215.5M
Coffee web‑shop order desk through a four‑act Christmas peakSmall businessMulti‑sessionScriptedSimulator207–2135 × 190%94%–0 of 5–306158.9M
Hospital assistant
Three‑day hospital ward care coordination, notes‑only memoryHospital72 simulated hoursScriptedSimulator290–4156 × 195%100%96%1 of 6–13554.1M
Radiation‑dose registry program from a long specHospital softwareSingle sessionNoneHidden tests46612 × 1100%100%100%8 of 12–142.7M
Emergency patient‑history compiler from a long specHealth records softwareSingle sessionNoneHidden tests9196 × 199%100%–2 of 6–3615.6M
Facilities engineer
Server‑room cooling outage in a 3D building with model‑played staffFacilitiesSingle sessionLLM‑voicedScript + LLM judges317 × 15%33%–0 of 792%61.6M
Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilitiesSingle sessionLLM‑voicedScript + LLM judges207 × 135%50%–0 of 795%61.4M
800 kg air handler move proved by trial in a 3D physics modelFacilitiesSingle sessionScriptedScript187 × 133%44%–0 of 7100%2912.7M
Engineering assistant
Scanned P&ID set digitised into instrument, line and equipment listsEngineering recordsSingle sessionScriptedScript3,8365 × 199%100%–1 of 5–171.3M
Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering recordsSingle sessionScriptedScript3,8365 × 199%99%–0 of 5–182.6M
P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering recordsSingle sessionScriptedScript656 × 197%100%–1 of 6–114.2M
Watchdog program commissioned for a running solvent recovery unitProcess controlCaveat: An idle agent earns 12%, though the grader fails itIn‑world clockScriptedSimulator416 × 179%93%–0 of 6–285.4M
DIY assistant
Room furnishing from photos: measure, match look‑alikes, order and planFurnishing serviceSingle sessionNoneScript288 × 363%80%–0 of 2499%1610.3M
Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacementSingle sessionNoneScript258 × 375%85%–0 of 24100%1315.8M
Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove managementSingle sessionNoneScript458 × 383%92%–0 of 24100%1912.8M
DIY smart‑lighting plan from notes, photos and one shop's catalogueHouseholdSingle sessionLLM‑voicedScript + LLM judges268 × 189%94%–0 of 8100%7917k
Night repair of a leaking condo floor on an in‑world clockHouseholdIn‑world clockNoneScript + LLM judges137 × 341%100%–1 of 21100%2254k
Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solarSingle sessionNoneSimulator156 × 171%99%–0 of 6–374.3M
Robot operator
Flat‑beam transfer by a bimanual robot with friction‑only gripsRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–2–
Three angled boards turned to flat outlines by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–8–
Two boards placed together by a bimanual robot, one per handRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–12–
Screwed two‑board lap joint by a bimanual robot with a hand‑held drillRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Screwed three‑board corner frame by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Screwed three‑board bridge by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Screwed five‑board grid from angled stock by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Screwed nine‑board grid with long rails by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Upright slatted bench from 11 boards by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Dog‑house frame and roof from 19 boards by a bimanual robotRobot manipulationSingle sessionNoneScript11 × 10%0%–0 of 1–20–
Baseball batting by a bimanual robot from batter‑eye imagesVisual robot manipulationSingle sessionNoneScriptPoints4 × 10%0%–––––
Race team
Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineeringSingle sessionScriptedScript45 × 14.6% slower2.0% slower–0 of 5–1920.0M
Four‑car race driven from cockpit images on a live physics clockMotor racingIn‑world clockNoneScript14 × 1100%100%–4 of 4–12–
Four‑car race on a live physics clock with lasting crash damageMotor racingIn‑world clockNoneScript14 × 1100%100%–3 of 4–18–
Setting
Where the job happens. Several environments share one: eleven are at one bakery chain and five at one coffee roaster, each with a different job.
Checks
The requirements the grader checks, each earning part of the reward; a range where the count varies by attempt.
Attempts
Models × attempts per model, with only the request; a range where models made different numbers.
Median, best
The median and the best attempt's reward, from 0 to 100%; for the race car, how much slower the car lapped than the fastest achievable car. More attempts give the best more chances.
Best open
The best attempt by an open‑weight model, where one was tested.
Solved
Full solves: attempts that met every requirement; for the race car, the fastest achievable car itself.
Told the rules
The best attempt when the request also gave the rules or method the job depends on; in 11 environments also facts from the files or people, in five of them figures the grader checks. A dash: not tested.
Minutes, tokens
The median attempt's wall‑clock time, over every attempt, and its tokens, cached input included; for tokens, where each model made several attempts, the median of the models' means.
Cost model
The reward prices the agent's plan from a baseline (0) to the best of tens of thousands of candidates (1), times the share of key facts quoted. In the bulk order, profit is 70%, up to the best outcome its staff can carry out, and checks 30%, less penalties.
Caveat
A known weakness in an environment's grader or setup, under its name. Methods lists them all.
Percentages
Rounded to the nearest point, except that only a reward of 1 shows as 100%; the race car's to one decimal.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.