Unstated requirements
Requests rarely say everything a job needs. Doing well means following the firm's conventions, finding the needs in the records, and doing what any professional would, unasked.
Three things an agent has to get right
Follow the firm's conventions
How the firm writes letters, notes and reports shows in its past work and later feedback, and the latest word beats an older template.
Find the needs in the records
Commitments, rules, limits and a family's standing needs sit in files, portal pages and profiles, and the request never spells them out.
Do what a professional would
Unrequested basics are checked too: sign‑in and backups in an app, isolating a unit before work, what makes a repair last.
They miss the firm's conventions, needs buried in the records, and professional basics
Conventions missed
Stranded‑family rebooking: every note missed habits shown only in the assistant's past trip summaries, such as writing times in 24‑hour form with the city.
Brewery input‑price report in house format: 75% of attempts judged how last month's recommendations went, as the outdated template asks, though the director had since objected.
Needs in the records missed
Course registration against rules found in portal: 50% of attempts kept a class that starts before the student's car park opens, a limit recorded only on the portal's pages.
Two‑site spares pick lists and stock plans: 67% of attempts picked the wrong parts for a two‑bearing failure at least once, though the work logs show it takes a shaft kit.
Professional basics missed
Server‑hall cooling diagnosis: no attempt proved the cause of the temperature swings by covering the gap behind each unit in turn and watching the swing settle; the brief named no method.
Night repair of a leaking condo floor: in 67% of attempts, the new floor wasn't flat and flush six weeks on; the brief asked for a repair that lasts, without saying how.
Partial credit for nearly every attempt, and few full solves
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Cancelled‑flight family rebooking against a ticking clockTravel concierge | Conventions. Five note habits appear only in the firm's past trip notes and carry 30% of the weight, scored per instance. | 61–63 | 8 | 85% |
| Household bill review: card disputes, claims, trackerHousehold bills | Conventions. Four letter habits appear only in the firm's latest letters, since the template on file is older. | 191–262 | 8 | 94% |
| Onboarding bill review with a simulated clientHousehold bills | Conventions. Six habits, in letters, the tracker and the note, appear only in the firm's latest past letters. | 295–320 | 8 | 84% |
| Medical bills review across two health plansHousehold bills | Conventions. The firm's practices and habits appear only in five past review packs of other households. | 287–350 | 8 | 88% |
| Course registration against rules found in portalUniversity portal | In the records. The request only says commitments exist; work shifts, training, the car park's hours and scholarship terms sit in portal pages. | 14 | 16 | 100% |
| Course registration optimised for earliest graduationUniversity portal | In the records. The same commitments, parking limits, scholarship terms and degree rules must be found across the portal's pages. | 14 | 15 | 93% |
| Stranded‑family rebooking with minors' exit papers and a dogPersonal assistant | Conventions. Four note habits appear only in the assistant's past trip summaries, such as giving times in 24‑hour form with the city. | 34 | 8 | 96% |
| Monthly waste report in house formatSmall business | Conventions. 23 checks test the format the owner now expects, learnable only from past reports and her feedback. | 43 | 13 | 89% |
| Brewery input‑price report in house formatSmall business | Conventions. 28 checks test the director's current format, from the total in the first sentence to no supplier names. | 51 | 12 | 90% |
| Contractor invoice audit against shop recordsSmall business | Conventions. How approvals, contractor letters and the payment note are done shows only in past approvals and managers' emails. | 107 | 13 | 61% |
| Bakery wholesale pack with downstream figure checksSmall business | Conventions. The pack must be built the house way from past packs, a style guide and the owner's latest word on totals. | 23 | 13 | 90% |
| Temperature compliance register from messy logger dataSmall business | Conventions. Past registers show the firm's breach rules, such as a short dip back in range not ending an excursion, and these beat a consultant's. | 103 | 12 | 40% |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | In the records. The request asks only for the form and a note; the policy's limits and extensions and the lease's notice deadline decide checks. | 29 | 7 | 99% |
| Bill‑audit and claims app from household filesHousehold | Basics. Only a few plain lines are given; sign‑in, recovery, later imports and the owner's own claim tracking are checked anyway. | 108 | 8 | 61% |
| Refinance‑broker case app from a non‑coder's laptopSmall business | In the records. The request names no feature; needs appear only in the owner's tracker, chats and notes, beside basics like backups. | 51 | 8 | 90% |
| Broker app with autonomous lead‑messaging assistantSmall business | In the records. The assistant's duties, rate limits and lead rules live only in the owner's chats, emails and the vendors' documents. | 106 | 8 | 68% |
| Multi‑shop bakery ordering app from business recordsSmall business | In the records. Collection rules, daily limits, notice periods and holidays live only in the records, and restore and auto‑start are checked too. | 97 | 8 | 75% |
| Music‑teaching studio app: lessons, billing, mediaSmall business | In the records. Dated prices, family billing, cancellation terms and recording consent live only in 416 files. | 155–179 | 7 | 81% |
| Laundry‑shop operations app for owner, hosts, workersSmall business | In the records. Prices, agreements, building rules and care terms live only in 53 plain‑text files, and recovery is checked unasked. | 161 | 7 | 77% |
| Listening piano‑practice app: following, transcription, feedbackHousehold | In the records. Needs and limits come from family notes, a teacher's guide and the score files, not the request. | 64 | 7 | 58% |
| Gym operations app with three owner‑feedback repair roundsSmall business | In the records. 87 checks rest on facts only in the files, and the other 26 on professional defaults, such as moving a booking. | 113 | 7 | 86% |
| Unattended train‑ticket buyer for a rate‑limited booking siteHousehold | In the records. Seven checks rest on facts outside the request, such as the site's pause and login limits on its Terms page. | 26 | 8 | 100% |
| Smart‑home retrofit: purchase plan, wiring and local control softwareHousehold | Basics. Only 38 of 121 checks tie to the owner's message; the rest come from house rules, product sheets and basics any system has. | 121 | 7 | 92% |
| Offline fault‑finding tablet app for data‑centre cooling unitsFacilities | Basics. Only 3 of 79 requirements cite the request; the rest include basics such as cutting the power before any work on a unit. | 79 | 7 | 87% |
| Flat‑pack assembly manuals with 3D views from unlabelled photosHousehold | Basics. Over half the requirements aren't in the request, such as distinct part names, hidden parts and a wall‑fixing step. | 173 | 7 | 63% |
| Returns‑label request hiding unraised account problemsCustomer service | Conventions. Three habits for giving items, orders and charges appear only in 10 past closed cases. | 62 | 8 | 98% |
| Appliance repair triage with budgeted customer questionsCustomer service | In the records. Parts technicians actually fit beyond the manual's list appear only in past job records. | 80 | 5 | 100% |
| Retail escalation reply and case recordsCustomer service | Conventions. Habits for naming items, orders, earlier refunds and tax splits appear only in past closed cases. | 114–115 | 15 | 98% |
| Missing‑payments reply from error‑laden retail recordsCustomer service | Conventions. Four habits for naming items and orders, earlier refunds and tax splits show only in 10 past closed cases. | 61 | 8 | 99% |
| Furniture warranty claim from a simulated customer's photosCustomer service | Conventions. How the team honours a gift or quotes an out‑of‑warranty part, and four writing habits, appear only as patterns in past cases. | 56 | 8 | 95% |
| Multi‑turn holiday‑rental search chat with saved profilesCustomer service | In the records. Standing needs sit on saved profiles, never restated, and each search must carry them unless the customer sets one aside. | 25 | 6 | 88% |
| Two‑site spares pick lists and stock plansManufacturing | In the records. Which parts a repair really takes shows only in the work logs; the documented lists are right on 12 of 33 reports. | 35 | 6 | 100% |
| Tea‑factory ops chat: speak or stay silentSmall business | Conventions. The factory's own settings practice appears only in the past chat, and sometimes contradicts the guide. | 71 | 6 | 53% |
| Tea factory rooftop solar and battery design with its business caseEnergy engineering | Basics. The system's size, next season's load and when the season ends are never given, and the ten‑year value depends on all three. | 18 | 6 | 92% |
| Emergency patient‑history compiler from a long specHealth records software | In the records. Five answer rules are documented nowhere and survive only in 22 forms the retired system filled. | 919 | 6 | 100% |
| Server‑room cooling outage in a 3D building with model‑played staffFacilities | Basics. The brief gives the goal but no limits: a second unit's start surge trips a shared circuit, and the red sockets feed the UPS. | 31 | 7 | 33% |
| Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilities | Basics. The brief gives no method, yet proof means covering the gap behind each unit in turn and seeing the temperature swing settle. | 20 | 7 | 50% |
| 800 kg air handler move proved by trial in a 3D physics modelFacilities | Basics. The brief gives the goal and three conditions but no method; the workers' safe pushing and lifting limits have to be found. | 18 | 7 | 44% |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | In the records. The client's practice on which instruments get a control‑system point is found only by asking its priced query desk. | 3,836 | 5 | 100% |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | In the records. The same client practice, found only through the priced query desk, decides which instruments get a control‑system point. | 3,836 | 5 | 99% |
| Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove management | Basics. Beyond the three placement standards the brief writes, layouts must keep pieces off heaters, out of door swings and clear of a child's windows. | 45 | 8 | 92% |
| DIY smart‑lighting plan from notes, photos and one shop's catalogueHousehold | Basics. The message gives wishes, not limits: beams must miss the reader's eyes and the fan's blades, and light the desk from in front. | 26 | 8 | 94% |
| Night repair of a leaking condo floor on an in‑world clockHousehold | Basics. The brief asks for a safe repair lasting about six weeks; checks spell out what that takes, such as leaving no wet wood. | 13 | 7 | 100% |
| Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineering | In the records. Six fast pairs of parts have broken down in past races, and only the series' past results show it. | 4 | 5 | 2.0% slower |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials