AsapiThe AI Agent Trials
Asapi / Capabilities / Unstated requirements
Capability · 44 environments

Unstated requirements

Requests rarely say everything a job needs. Doing well means following the firm's conventions, finding the needs in the records, and doing what any professional would, unasked.

What it takes

Three things an agent has to get right

Follow the firm's conventions

How the firm writes letters, notes and reports shows in its past work and later feedback, and the latest word beats an older template.

Find the needs in the records

Commitments, rules, limits and a family's standing needs sit in files, portal pages and profiles, and the request never spells them out.

Do what a professional would

Unrequested basics are checked too: sign‑in and backups in an app, isolating a unit before work, what makes a repair last.

What models get wrong

They miss the firm's conventions, needs buried in the records, and professional basics

Conventions missed

  1. Stranded‑family rebooking: every note missed habits shown only in the assistant's past trip summaries, such as writing times in 24‑hour form with the city.

  2. Brewery input‑price report in house format: 75% of attempts judged how last month's recommendations went, as the outdated template asks, though the director had since objected.

Needs in the records missed

  1. Course registration against rules found in portal: 50% of attempts kept a class that starts before the student's car park opens, a limit recorded only on the portal's pages.

  2. Two‑site spares pick lists and stock plans: 67% of attempts picked the wrong parts for a two‑bearing failure at least once, though the work logs show it takes a shaft kit.

Professional basics missed

  1. Server‑hall cooling diagnosis: no attempt proved the cause of the temperature swings by covering the gap behind each unit in turn and watching the swing settle; the brief named no method.

  2. Night repair of a leaking condo floor: in 67% of attempts, the new floor wasn't flat and flush six weeks on; the brief asked for a repair that lasts, without saying how.

Results

Partial credit for nearly every attempt, and few full solves

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

EnvironmentMedianFully solved
Cancelled‑flight family rebooking against a ticking clock8 models × 3 attempts70%0 of 24Household bill review: card disputes, claims, tracker8 models × 3 attempts86%0 of 24Onboarding bill review with a simulated client8 models × 3 attempts78%0 of 24Medical bills review across two health plans8 models × 3 attempts74%0 of 24Course registration against rules found in portal16 models71%2 of 16Course registration optimised for earliest graduation15 models79%0 of 15Stranded‑family rebooking with minors' exit papers and a dog8 models × 3 attempts61%0 of 24Monthly waste report in house format13 models64%0 of 13Brewery input‑price report in house format12 models73%0 of 12Contractor invoice audit against shop records13 models44%0 of 13Bakery wholesale pack with downstream figure checks13 models48%0 of 13Temperature compliance register from messy logger data12 models9%0 of 12Kitchen‑fire contents claim from stale counts, invoices and photos7 models81%0 of 7Bill‑audit and claims app from household files8 models57%0 of 8Refinance‑broker case app from a non‑coder's laptop8 models75%0 of 8Broker app with autonomous lead‑messaging assistant8 models58%0 of 8Multi‑shop bakery ordering app from business records8 models60%0 of 8Music‑teaching studio app: lessons, billing, media7 models76%0 of 7Laundry‑shop operations app for owner, hosts, workers7 models65%0 of 7Listening piano‑practice app: following, transcription, feedback7 models53%0 of 7Gym operations app with three owner‑feedback repair rounds7 models82%0 of 7Unattended train‑ticket buyer for a rate‑limited booking site8 models83%1 of 8Smart‑home retrofit: purchase plan, wiring and local control software7 models76%0 of 7Offline fault‑finding tablet app for data‑centre cooling units7 models64%0 of 7Flat‑pack assembly manuals with 3D views from unlabelled photos7 models53%0 of 7Returns‑label request hiding unraised account problems8 models × 3 attempts94%0 of 24Appliance repair triage with budgeted customer questions5 models100%3 of 5Retail escalation reply and case records15 models73%0 of 15Missing‑payments reply from error‑laden retail records8 models × 3 attempts78%0 of 24Furniture warranty claim from a simulated customer's photos8 models × 3 attempts82%0 of 24Multi‑turn holiday‑rental search chat with saved profiles6 models34%0 of 6Two‑site spares pick lists and stock plans6 models90%1 of 6Tea‑factory ops chat: speak or stay silent6 models11%0 of 6Tea factory rooftop solar and battery design with its business case6 models66%0 of 6Emergency patient‑history compiler from a long spec6 models99%2 of 6Server‑room cooling outage in a 3D building with model‑played staff7 models5%0 of 7Server‑hall cooling diagnosis by undoable tests in a 3D building7 models35%0 of 7800 kg air handler move proved by trial in a 3D physics model7 models33%0 of 7Scanned P&ID set digitised into instrument, line and equipment lists5 models99%1 of 5Photographed P&ID sketches digitised into instrument, line and equipment lists5 models99%0 of 5Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layouts8 models × 3 attempts83%0 of 24DIY smart‑lighting plan from notes, photos and one shop's catalogue8 models89%0 of 8Night repair of a leaking condo floor on an in‑world clock7 models × 3 attempts41%1 of 21Club race‑car design and parts procurement on a web app with hidden supplier terms5 models96%0 of 5

Hover a dot for its model and reward.

The 44 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Cancelled‑flight family rebooking against a ticking clockTravel conciergeConventions. Five note habits appear only in the firm's past trip notes and carry 30% of the weight, scored per instance.61–63885%
Household bill review: card disputes, claims, trackerHousehold billsConventions. Four letter habits appear only in the firm's latest letters, since the template on file is older.191–262894%
Onboarding bill review with a simulated clientHousehold billsConventions. Six habits, in letters, the tracker and the note, appear only in the firm's latest past letters.295–320884%
Medical bills review across two health plansHousehold billsConventions. The firm's practices and habits appear only in five past review packs of other households.287–350888%
Course registration against rules found in portalUniversity portalIn the records. The request only says commitments exist; work shifts, training, the car park's hours and scholarship terms sit in portal pages.1416100%
Course registration optimised for earliest graduationUniversity portalIn the records. The same commitments, parking limits, scholarship terms and degree rules must be found across the portal's pages.141593%
Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantConventions. Four note habits appear only in the assistant's past trip summaries, such as giving times in 24‑hour form with the city.34896%
Monthly waste report in house formatSmall businessConventions. 23 checks test the format the owner now expects, learnable only from past reports and her feedback.431389%
Brewery input‑price report in house formatSmall businessConventions. 28 checks test the director's current format, from the total in the first sentence to no supplier names.511290%
Contractor invoice audit against shop recordsSmall businessConventions. How approvals, contractor letters and the payment note are done shows only in past approvals and managers' emails.1071361%
Bakery wholesale pack with downstream figure checksSmall businessConventions. The pack must be built the house way from past packs, a style guide and the owner's latest word on totals.231390%
Temperature compliance register from messy logger dataSmall businessConventions. Past registers show the firm's breach rules, such as a short dip back in range not ending an excursion, and these beat a consultant's.1031240%
Kitchen‑fire contents claim from stale counts, invoices and photosSmall businessIn the records. The request asks only for the form and a note; the policy's limits and extensions and the lease's notice deadline decide checks.29799%
Bill‑audit and claims app from household filesHouseholdBasics. Only a few plain lines are given; sign‑in, recovery, later imports and the owner's own claim tracking are checked anyway.108861%
Refinance‑broker case app from a non‑coder's laptopSmall businessIn the records. The request names no feature; needs appear only in the owner's tracker, chats and notes, beside basics like backups.51890%
Broker app with autonomous lead‑messaging assistantSmall businessIn the records. The assistant's duties, rate limits and lead rules live only in the owner's chats, emails and the vendors' documents.106868%
Multi‑shop bakery ordering app from business recordsSmall businessIn the records. Collection rules, daily limits, notice periods and holidays live only in the records, and restore and auto‑start are checked too.97875%
Music‑teaching studio app: lessons, billing, mediaSmall businessIn the records. Dated prices, family billing, cancellation terms and recording consent live only in 416 files.155–179781%
Laundry‑shop operations app for owner, hosts, workersSmall businessIn the records. Prices, agreements, building rules and care terms live only in 53 plain‑text files, and recovery is checked unasked.161777%
Listening piano‑practice app: following, transcription, feedbackHouseholdIn the records. Needs and limits come from family notes, a teacher's guide and the score files, not the request.64758%
Gym operations app with three owner‑feedback repair roundsSmall businessIn the records. 87 checks rest on facts only in the files, and the other 26 on professional defaults, such as moving a booking.113786%
Unattended train‑ticket buyer for a rate‑limited booking siteHouseholdIn the records. Seven checks rest on facts outside the request, such as the site's pause and login limits on its Terms page.268100%
Smart‑home retrofit: purchase plan, wiring and local control softwareHouseholdBasics. Only 38 of 121 checks tie to the owner's message; the rest come from house rules, product sheets and basics any system has.121792%
Offline fault‑finding tablet app for data‑centre cooling unitsFacilitiesBasics. Only 3 of 79 requirements cite the request; the rest include basics such as cutting the power before any work on a unit.79787%
Flat‑pack assembly manuals with 3D views from unlabelled photosHouseholdBasics. Over half the requirements aren't in the request, such as distinct part names, hidden parts and a wall‑fixing step.173763%
Returns‑label request hiding unraised account problemsCustomer serviceConventions. Three habits for giving items, orders and charges appear only in 10 past closed cases.62898%
Appliance repair triage with budgeted customer questionsCustomer serviceIn the records. Parts technicians actually fit beyond the manual's list appear only in past job records.805100%
Retail escalation reply and case recordsCustomer serviceConventions. Habits for naming items, orders, earlier refunds and tax splits appear only in past closed cases.114–1151598%
Missing‑payments reply from error‑laden retail recordsCustomer serviceConventions. Four habits for naming items and orders, earlier refunds and tax splits show only in 10 past closed cases.61899%
Furniture warranty claim from a simulated customer's photosCustomer serviceConventions. How the team honours a gift or quotes an out‑of‑warranty part, and four writing habits, appear only as patterns in past cases.56895%
Multi‑turn holiday‑rental search chat with saved profilesCustomer serviceIn the records. Standing needs sit on saved profiles, never restated, and each search must carry them unless the customer sets one aside.25688%
Two‑site spares pick lists and stock plansManufacturingIn the records. Which parts a repair really takes shows only in the work logs; the documented lists are right on 12 of 33 reports.356100%
Tea‑factory ops chat: speak or stay silentSmall businessConventions. The factory's own settings practice appears only in the past chat, and sometimes contradicts the guide.71653%
Tea factory rooftop solar and battery design with its business caseEnergy engineeringBasics. The system's size, next season's load and when the season ends are never given, and the ten‑year value depends on all three.18692%
Emergency patient‑history compiler from a long specHealth records softwareIn the records. Five answer rules are documented nowhere and survive only in 22 forms the retired system filled.9196100%
Server‑room cooling outage in a 3D building with model‑played staffFacilitiesBasics. The brief gives the goal but no limits: a second unit's start surge trips a shared circuit, and the red sockets feed the UPS.31733%
Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilitiesBasics. The brief gives no method, yet proof means covering the gap behind each unit in turn and seeing the temperature swing settle.20750%
800 kg air handler move proved by trial in a 3D physics modelFacilitiesBasics. The brief gives the goal and three conditions but no method; the workers' safe pushing and lifting limits have to be found.18744%
Scanned P&ID set digitised into instrument, line and equipment listsEngineering recordsIn the records. The client's practice on which instruments get a control‑system point is found only by asking its priced query desk.3,8365100%
Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering recordsIn the records. The same client practice, found only through the priced query desk, decides which instruments get a control‑system point.3,836599%
Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove managementBasics. Beyond the three placement standards the brief writes, layouts must keep pieces off heaters, out of door swings and clear of a child's windows.45892%
DIY smart‑lighting plan from notes, photos and one shop's catalogueHouseholdBasics. The message gives wishes, not limits: beams must miss the reader's eyes and the fan's blades, and light the desk from in front.26894%
Night repair of a leaking condo floor on an in‑world clockHouseholdBasics. The brief asks for a safe repair lasting about six weeks; checks spell out what that takes, such as leaving no wet wood.137100%
Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineeringIn the records. Six fast pairs of parts have broken down in past races, and only the series' past results show it.452.0% slower

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.