AsapiFor AI labs / Catalogue
56 environments, hardest first
One row per environment: how it works, how it's graded, and how the models we tested did with only the request, as a user would give it. Sorted by the best attempt, lowest first. Filter by capability.
| Environment | Time | People | Grader | Checks | Attempts | Median | Best | Best open | Solved | Told the rules | Minutes | Tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Listening piano‑practice app: following, transcription, feedbackA three‑player piano family practising at home at beginner, intermediate and advanced levels | Single session | None | Tester model + judges | 64 | 7 × 1 | 53% | 58% | – | 0 of 7 | 62% | 41 | 5.3M |
| Bill‑audit and claims app from household filesA household after a move, with card statements, bills, receipts and emails in disarray | Single session | None | Tester model + judges | 108 | 8 × 1 | 57% | 61% | – | 0 of 8 | 76% | 14 | 3.2M |
| Flat‑pack assembly manuals with 3D views from unlabelled photosA home user in a new flat with 10 flat‑pack furniture pieces, no instruction booklets and a seller who does not answer | Single session | None | Tester model + judges | 173 | 7 × 1 | 53% | 63% | – | 0 of 7 | 64% | 33 | 3.7M |
| Tea‑factory ops chat: speak or stay silentOperations chat of a factory making black tea (withering and drying)Caveat: Posting nothing scores 50%, since the score averages speaking when needed and staying silent | Short check‑in sessions | None | Script | 106 | 6 × 1 | 49% | 67% | 44% | 0 of 6 | – | 172 | 14.3M |
| Multi‑turn holiday‑rental search chat with saved profilesChat desk of a holiday‑rental search service over a messy multi‑source listings databaseCaveat: Our attempts weren't isolated from the answer key | Single session | Scripted | Script | 25 | 6 × 1 | 40% | 68% | – | 0 of 6 | – | 23 | 18.0M |
| Broker app with autonomous lead‑messaging assistantThe same home‑loan refinancing side business, now signed up to a chat‑messaging business sandbox and two property‑data services | Single session | None | Tester model + judges | 106 | 8 × 1 | 58% | 68% | – | 0 of 8 | 91% | 19 | 2.6M |
| Server‑hall cooling diagnosis by undoable tests in a 3D buildingAn FM company's on‑site engineer in a 3D model of a police station, whose two new server‑hall cooling units have swung all day since a contractor replaced them three weeks earlier | Single session | LLM‑voiced | Script + LLM judges | 20 | 7 × 1 | 55% | 70% | – | 0 of 7 | 95% | 6 | 1.4M |
| Multi‑shop bakery ordering app from business recordsA cake bakery with several shops and a central kitchen whose customers want to order online | Single session | None | Tester model + judges | 97 | 8 × 1 | 60% | 75% | – | 0 of 8 | 99% | 7 | 1.5M |
| Temperature compliance register from messy logger dataSmall business (bakery chain with six shops) | Single session | None | Script | 103 | 12 × 1 | 42% | 76% | 41% | 0 of 12 | 95% | 46 | 12.6M |
| Laundry‑shop operations app for owner, hosts, workersA small laundry shop serving short‑let hosts and walk‑in customers, with hourly collection rounds and delivery workers | Single session | None | Tester model + judges | 161 | 7 × 1 | 65% | 77% | – | 0 of 7 | 81% | 39 | 4.7M |
| Live server‑room cooling incident in a 3D building with model‑played staffA police station's facilities duty engineer (FM company), in a 3D model of the station, at the alarm that both live server‑room cooling units have stopped | Single session | LLM‑voiced | Script + LLM judges | 31 | 7 × 1 | 65% | 77% | – | 0 of 7 | 100% | 11 | 1.8M |
| Household solar purchase plan with simulated contactsHousehold (family home; the owner's brother pays the power bill) | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 82 | 10 × 1 | 56% | 78% | 71% | 0 of 10 | 100% | 12 | 1.6M |
| Course registration against rules found in portalUniversity course‑registration web portalCaveat: An empty registration meets 11 of 14 checks | Single session | None | Script | 14 | 4 × 1 | 79% | 79% | – | 0 of 4 | – | 3 | 333k |
| Course registration optimised for earliest graduationUniversity course‑registration web portalCaveat: An empty registration meets 11 of 14 checks | Single session | None | Script | 14 | 1 × 1 | 79% | 79% | – | 0 of 1 | – | 12 | 899k |
| Exact holiday‑rental retrieval over multilingual multi‑source dataHoliday‑rental search service with a listings database split across feeds, a licence register and map layersCaveat: Our attempts weren't isolated from the answer key | Single session | None | Script | 20 | 6 × 1 | 80% | 80% | – | 0 of 6 | – | 14 | 3.4M |
| Room furnishing from photos: measure, match look‑alikes, order and planA furnishing concierge's assistant, with the firm's trade accounts at three furniture web shops, on 1 February 2027, furnishing a client's flat like the building's staged model apartment | Single session | None | Script | 28 | 8 × 3 | 63% | 80% | – | 0 of 24 | 99% | 16 | 10.3M |
| Music‑teaching studio app: lessons, billing, mediaA self‑employed piano and theory teacher's home studio, with pupils, parents and an accountant | Single session | None | Tester model + judges | 155–179 | 7 × 1 | 76% | 81% | – | 0 of 7 | 94% | 20 | 9.0M |
| Onboarding bill review with a simulated clientThe same bill‑management firm's review desk; a new client's first review, with a client message thread | Single session | LLM‑voiced | Script + LLM judges | 295–320 | 8 × 3 | 78% | 84% | – | 0 of 24 | 95% | 10 | 10.9M |
| Cancelled‑flight family rebooking against a ticking clockA travel concierge's trip desk on the evening a family's overnight flight to a wedding abroad is cancelledCaveat: A single read‑only lookup scores about 20% | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 61–63 | 8 × 3 | 70% | 85% | – | 0 of 24 | 99% | 10 | 10.5M |
| Gym operations app with three owner‑feedback repair roundsAn independent gym in an office tower: the owner coaches 22 people, with three freelance trainers, a front desk and a staff contract just signed with the bank upstairs | Single session | LLM‑voiced | Tester model + judges | 113 | 7 × 1 | 82% | 86% | – | 0 of 7 | 86% | 35 | 9.2M |
| Family dinner booking with simulated relativesHousehold (extended family of seven households) | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 93 | 9 × 1 | 7% | 86% | 9% | 0 of 9 | 95% | 18 | 1.8M |
| Offline fault‑finding tablet app for data‑centre cooling unitsA data‑centre hall with 12 cooling units of three models, after a night shift worked from the wrong drawing and a contractor bridged a high‑pressure switch | Single session | None | Tester model + judges | 79 | 7 × 1 | 64% | 87% | – | 0 of 7 | 83% | 27 | 5.8M |
| Medical bills review across two health plansThe same firm's review desk; a couple's medical and pharmacy bills for a plan year, with two employer plans, a pharmacy benefit, a flexible spending account and a card | Single session | LLM‑voiced | Script + LLM judges | 287–350 | 8 × 3 | 74% | 88% | – | 0 of 24 | 99% | 16 | 14.3M |
| Seasonal bulk order run with simulated staffSmall business (bakery chain supplying a hotel group) | In‑world clock | LLM‑voiced + scripted | Cost model + LLM judges | 23 | 8 × 1 | 48% | 88% | – | 0 of 8 | 96% | 17 | 1.9M |
| Bakery's freezer‑loss insurance claim packSmall business (bakery chain with six shops) | Single session | None | Script + LLM judges | 60 | 14 × 1 | 66% | 89% | 89% | 0 of 14 | 100% | 11 | 1.5M |
| Household savings plan from records and familyHousehold (multi‑generation family home) | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 51 | 8 × 1 | 73% | 89% | – | 0 of 8 | 98% | 8 | 1.5M |
| Contractor invoice audit against shop recordsSmall business (bakery chain with six shops) | Single session | None | Script + LLM judges | 107 | 13 × 1 | 67% | 90% | 76% | 0 of 13 | 100% | 15 | 2.7M |
| Refinance‑broker case app from a non‑coder's laptopA one‑person home‑loan refinancing side business run from a personal laptop | Single session | None | Tester model + judges | 51 | 8 × 1 | 75% | 90% | – | 0 of 8 | 98% | 19 | 2.8M |
| Brewery input‑price report in house formatSmall business (brewery)Caveat: A bare table with its total, one note and a recommendation scores 97% | Single session | None | Script + LLM judges | 51 | 12 × 1 | 77% | 90% | 81% | 0 of 12 | 100% | 8 | 1.0M |
| Bakery wholesale pack with downstream figure checksSmall business (bakery chain) | Single session | None | Script + LLM judges | 23 | 13 × 1 | 56% | 91% | 58% | 0 of 13 | 100% | 13 | 1.7M |
| Smart‑home retrofit: purchase plan, wiring and local control softwareA small, old terrace house with no smart setup, an always‑on Mac mini, two iPads, a phone and an electrician who wires only from a table | Single session | None | Tester model + judges | 121 | 7 × 1 | 76% | 92% | – | 0 of 7 | 96% | 32 | 11.3M |
| Coffee web‑shop order desk through a four‑act Christmas peakOrder desk of a coffee roaster's web shop, subscriptions and trade counter, with two carriers, a payment gateway, a warehouse, a roastery and a helpdeskCaveat: Doing nothing scores about half, from restraint checks, though every act fails | Multi‑session | Scripted | Simulator | 207–213 | 5 × 1 | 90% | 94% | – | 0 of 5 | – | 306 | 158.9M |
| Packaging proof sign‑off for an absent ownerSmall business (bakery chain) | Single session | None | Script + LLM judges | 15 | 13 × 1 | 27% | 94% | 27% | 0 of 13 | 100% | 7 | 1.7M |
| Monthly waste report in house formatSmall business (bakery chain with six shops)Caveat: A figures‑only report scores 84% | Single session | None | Script + LLM judges | 43 | 13 × 1 | 75% | 94% | 82% | 0 of 13 | 100% | 7 | 1.4M |
| Household bill review: card disputes, claims, trackerThe review desk of a household bill‑management firm, worked through its workspace of tickets, wiki and client vault | Single session | None | Script + LLM judges | 191–262 | 8 × 3 | 86% | 94% | – | 0 of 24 | – | 12 | 17.8M |
| Stranded‑family rebooking with minors' exit papers and a dogA father's personal assistant, working from the family's hub, on the Sunday evening the airline cancels the mother's, two children's and dog's flight home from abroad and moves them to ThursdayCaveat: A legal early flight chosen without asking about the papers scores about 90% | Single session | LLM‑voiced | Script + LLM judges | 34 | 8 × 3 | 61% | 96% | – | 0 of 24 | 99% | 9 | 8.7M |
| Coffee roaster materials planning across two seasonsMaterials planning at a coffee roaster with contract customers, green‑coffee importers, packaging makers and a shipping lineCaveat: Claude Code's 4 attempts ran outside a container, beside the simulator and grader; transcripts were checked afterwards | Two 16‑week windows | Scripted | Simulator | 114–168 | 5 × 1–2 | 90% | 97% | – | 0 of 7 | – | 113 | 43.9M |
| Transit journey planner and phone telemetry over five field weeksEngineering of a public‑transport journey‑planning app for a made‑up city's metro, tram, bus, express‑bus and ferry network, used between releases by 120 simulated riders | Multi‑session | Scripted | Script | 81 | 5 × 1 | 84% | 98% | – | 0 of 5 | – | 115 | 23.8M |
| Retail escalation reply and case recordsThe escalations desk of a home‑goods retailer, worked through its customer‑service hub | Single session | None | Script + LLM judges | 114–115 | 15 × 1 | 73% | 98% | 30% | 0 of 15 | 100% | 6 | 2.8M |
| Returns‑label request hiding unraised account problemsThe same retailer's escalations desk; a customer who asks only for a new returns label | Single session | None | Script + LLM judges | 62 | 8 × 3 | 94% | 98% | – | 0 of 24 | 100% | 4 | 1.6M |
| Three‑year repair‑or‑replace plan with simulated managerSmall business (bakery chain; one shop's equipment) | Single session | LLM‑voiced | Cost model | Cost model | 12 × 1 | 68% | 99% | 82% | 0 of 12 | 99% | 8 | 181k |
| Missing‑payments reply from error‑laden retail recordsThe same retailer's escalations desk; a customer chasing four payments she says never reached her | Single session | None | Script + LLM judges | 61 | 8 × 3 | 78% | 99% | – | 0 of 24 | 100% | 5 | 2.3M |
| 12‑month equipment plan with simulated shop managerSmall business (bakery chain; one shop's equipment) | Single session | LLM‑voiced | Cost model | Cost model | 13 × 1 | 0% | 100% | 33% | 1 of 13 | 100% | 3 | 82k |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business: a bakery chain, one shop's back kitchen the morning after a fire | In‑world clock | LLM‑voiced + scripted | Script + LLM judges | 29 | 7 × 1 | 86% | 100% | – | 1 of 7 | 100% | 11 | 3.0M |
| Unattended train‑ticket buyer for a rate‑limited booking siteA weekly commuter who goes home by train every weekend, on sold‑out trains, with one account note on the laptop | Single session | None | Tester model + judges | 26 | 8 × 1 | 83% | 100% | – | 1 of 8 | 100% | 12 | 2.0M |
| Emergency patient‑history compiler from a long specRegional health‑information exchange serving emergency departments, built as software in a Linux container | Single session | None | Hidden tests | 919 | 6 × 1 | 99% | 100% | – | 2 of 6 | – | 36 | 15.6M |
| Appliance repair triage with budgeted customer questionsAppliance repair business's customer desk, earning only on parts and labour | 80 separate sessions | Scripted | Script | 80 | 5 × 1 | 100% | 100% | – | 3 of 5 | – | 94 | 37.7M |
| Radiation‑dose registry program from a long specMedical‑imaging radiation‑dose registry, built as software in a Linux container | Single session | None | Hidden tests | 466 | 12 × 1 | 100% | 100% | 100% | 8 of 12 | – | 14 | 2.7M |
| Month‑long parcel fleet dispatch, notes‑only memoryCity parcel carrier with three hubs and a mixed van and box‑truck fleet | 28 simulated days | Scripted | Simulator | 29 | 2 × 1 | 62% | 100% | – | 1 of 2 | – | 158 | 52.7M |
| Tea‑factory cash desk over thirteen weekly sessionsFinance desk of a tea factory that sells to trade buyers and at auction, with a bank facility and an owner who can lend | 13 weekly sessions | Scripted | Simulator | 64 | 5 × 1 | 81% | 100% | – | 1 of 5 | – | 559 | 129.5M |
| Packing‑line project purchasing over twelve mail‑driven weeksNew packing‑line installation at a coffee roaster's site, with a design consultancy, a machine maker and an electrical contractor | 12 simulated weeks | Scripted | Simulator | 14 | 6 × 3 | 100% | 100% | – | 13 of 18 | – | 37 | 9.3M |
| P&ID as‑built backlog from nine teams' records, drafted in DXFDrawing office for a made‑up process plant unit (two trains, cooling water, a flare knock‑out drum) whose P&IDs, held as a nine‑sheet DXF CAD master, have fallen six months behind the plant; the agent is a contract drafter | Single session | Scripted | Script | 65 | 6 × 1 | 97% | 100% | – | 1 of 6 | – | 11 | 4.2M |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records for a made‑up process plant unit (two process trains, cooling water, a flare knock‑out drum) with no CAD source; the agent is a digitisation consultant on a fixed price | Single session | Scripted | Script | 3,836 | 5 × 1 | 99% | 100% | – | 1 of 5 | – | 17 | 1.3M |
| Two‑site spares pick lists and stock plansSpare‑parts stores at a coffee roaster with two sitesCaveat: An idle agent earns 18%, though the grader fails it | 8 weekly sessions | None | Simulator | 35 | 6 × 1 | 97% | 100% | – | 1 of 6 | – | 22 | 4.5M |
| Two‑site spares buying through supplier part withdrawalsPurchasing for the spare‑parts stores of a two‑site coffee roasterCaveat: An idle agent earns 39%, though the grader fails it | 12 simulated weeks | Scripted | Simulator | 62 | 5 × 1 | 98% | 100% | – | 2 of 5 | – | 61 | 15.4M |
| Three‑day hospital ward care coordination, notes‑only memorySimulated hospital ward with five patients, nurses on day and night shifts, physicians and a diagnostics department | 72 simulated hours | Scripted | Simulator | 290–415 | 6 × 1 | 98% | 100% | 97% | 1 of 6 | – | 135 | 54.1M |
- Setting
- Where the job happens. Several environments share one: eleven are at one bakery chain and five at one coffee roaster, each with a different job.
- Checks
- The requirements the grader checks, each earning part of the reward; a range where the count varies by attempt.
- Attempts
- Models × attempts per model, with only the request; a range where models made different numbers.
- Median, best
- The median and the best attempt's reward, from 0 to 100%. More attempts give the best more chances.
- Best open
- The best attempt by an open‑weight model, where one was tested.
- Solved
- Full solves: attempts that met every requirement.
- Told the rules
- The best attempt when the request also gave the rules or method the job depends on; in 7 environments also facts from the files or people, in three of them figures the grader checks. A dash: not tested.
- Minutes, tokens
- The median attempt's wall‑clock time, over every attempt, and its tokens, cached input included; for tokens, where each model made several attempts, the median of the models' means.
- Cost model
- The reward prices the agent's plan, from a baseline plan (0) to the best of tens of thousands of candidates a search found (1). In the seasonal bulk order, profit is 70%, up to the best outcome its staff can carry out; checks give 30%, less penalties.
- Caveat
- A known weakness in an environment's grader or setup, under its name. Methods lists them all.
- Percentages
- Rounded to the nearest point, except that only a reward of 1 shows as 100%.
Try a sample environment
Three are downloadable as Harbor tasks, with graders and reference solutions. Request one with your work email, or book a call about the other 53.