Catching errors
Records hold mistakes, equipment hides faults, and sources disagree. Doing well means spotting what's wrong, deciding which source to trust, and leaving what's right alone.
Three things an agent has to get right
Spot what's wrong
Planted mistakes and faults go unmentioned, such as altered payslips, a wrongly charted dose or a disconnected fire‑suppression cylinder.
Decide which source to trust
Superseded reports, edited records, old invoices and mistaken beliefs contradict the facts, so the agent has to judge which source holds.
Leave what's right alone
Look‑alikes, genuine oddities and correct lines sit among the errors, and in many jobs, flagging them costs reward.
They miss planted errors, trust the wrong record, or raise false alarms
Errors missed
Bakery wholesale pack with downstream figure checks: 85% of attempts never tested another team's ordering formula on months of different lengths, so the error planted in it went unreported.
Server‑room cooling outage: 43% of attempts handed in their report with a fire‑suppression cylinder still disconnected, a fault shown only on its panel.
The wrong source trusted
Bakery's freezer‑loss insurance claim pack: 79% of attempts used a disposal record that had been edited later, though the same‑day evidence contradicts it.
Kitchen‑fire contents claim: 57% of attempts claimed the worktables and baking trays from old invoices rather than what stood in the kitchen on the night.
False alarms
Temperature compliance register: 58% of attempts handed in a register with invented extra breaches, such as a planned deep clean logged as one.
Watchdog program for a running plant: 33% of attempts cut the heater on a one‑point temperature spike, treating a harmless blip as a fault.
Partial credit for nearly every attempt, and few full solves
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Cancelled‑flight family rebooking against a ticking clockTravel concierge | Spotting. The bookings hold problems the family never raised, such as a schedule change on another leg, or seats lost when a ticket was reissued. | 61–63 | 8 | 85% |
| Household bill review: card disputes, claims, trackerHousehold bills | Leaving alone. Wrong charges hide among legitimate look‑alikes, partial credits and promised refunds, and disputing a look‑alike costs reward. | 191–262 | 8 | 94% |
| Onboarding bill review with a simulated clientHousehold bills | Which source. Statement names differ from shop names, companies were renamed, and refund messages must be checked against the account. | 295–320 | 8 | 84% |
| Medical bills review across two health plansHousehold bills | Spotting. Problems nobody flagged carry 22% of the weight and must be found record by record, under both health plans' rules. | 287–350 | 8 | 88% |
| Course registration against rules found in portalUniversity portal | Spotting. The starting registration holds problem sections the portal accepts silently, such as a class before the car park opens. | 14 | 16 | 100% |
| Course registration optimised for earliest graduationUniversity portal | Spotting. Similar starting problems, such as a course without its prerequisite that also meets on work days, have to be found and fixed. | 14 | 15 | 93% |
| Stranded‑family rebooking with minors' exit papers and a dogPersonal assistant | Which source. The father believes the mother is on the rental contract; she isn't, and only the contract itself shows it. | 34 | 8 | 96% |
| Packaging proof sign‑off for an absent ownerSmall business | Which source. A supplier's stated size for a swapped part conflicts with the maker's own sheet, and designs were re‑lettered between rounds. | 15 | 13 | 84% |
| Monthly waste report in house formatSmall business | Leaving alone. Flagging recoverable data as missing costs reward, as does missing a real gap, a double‑counted till extract or the wrong denominator. | 43 | 13 | 89% |
| Brewery input‑price report in house formatSmall business | Spotting. A returned load, a double scan beside a genuine look‑alike, a withdrawn surcharge and an unpriced lot all have to be handled. | 51 | 12 | 90% |
| Contractor invoice audit against shop recordsSmall business | Leaving alone. Planted over- and under‑charges sit among 44 invoice lines, and querying a correct line fails a check. | 107 | 13 | 61% |
| Bakery's freezer‑loss insurance claim packSmall business | Which source. A draft with mistakes, a superseded engineer's report and an edited disposal record conflict with same‑day evidence. | 60 | 14 | 83% |
| Bakery wholesale pack with downstream figure checksSmall business | Spotting. Planted problems in last month's figures, a customer's rebate and another team's formula must be found, without false alarms. | 23 | 13 | 90% |
| Temperature compliance register from messy logger dataSmall business | Leaving alone. Genuine unusual entries must be trusted and extra breaches never invented, while faulty sensors, clock errors and a fabricated sheet are caught. | 103 | 12 | 40% |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | Which source. Records true once but not on the night decide five checks, such as a flour count taken before six sacks left. | 29 | 7 | 99% |
| Bill‑audit and claims app from household filesHousehold | Leaving alone. The app must find wrong charges and route each claim, while pending holds, real repeats and refunds already received are left alone. | 108 | 8 | 61% |
| Refinance‑broker case app from a non‑coder's laptopSmall business | Leaving alone. Genuine but messy documents mustn't be marked suspect, while those with figures the grader altered must be flagged. | 51 | 8 | 90% |
| Broker app with autonomous lead‑messaging assistantSmall business | Spotting. As in the refinance‑broker app, statements and payslips with altered or inconsistent figures must be flagged. | 106 | 8 | 68% |
| Listening piano‑practice app: following, transcription, feedbackHousehold | Spotting. The app's mistake report must list planted wrong, missed, extra and late notes at their bar and beat, and none on a clean take. | 64 | 7 | 58% |
| Gym operations app with three owner‑feedback repair roundsSmall business | Which source. Old records are wrong or duplicated: balances must be rebuilt from dated events, and same‑name strangers kept apart. | 113 | 7 | 86% |
| Offline fault‑finding tablet app for data‑centre cooling unitsFacilities | Spotting. A blown fuse usually hides a shorted coil or winding, and some causes show only in the alarm history or maintenance log. | 79 | 7 | 87% |
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Which source. Nearly half the graded requests turn on places where the operator's timetable feed is wrong or silent. | 81 | 5 | 98% |
| Returns‑label request hiding unraised account problemsCustomer service | Spotting. Problems the customer never raised sit in her account, and a colleague's unsent draft has the label against the wrong return. | 62 | 8 | 98% |
| Retail escalation reply and case recordsCustomer service | Which source. The case file holds misfiled documents, three policy versions and a colleague's draft with wrong figures. | 114–115 | 15 | 98% |
| Missing‑payments reply from error‑laden retail recordsCustomer service | Spotting. Each payment the customer chases turns on a different record error or on a claim the records don't support, and paying twice fails checks. | 61 | 8 | 99% |
| Furniture warranty claim from a simulated customer's photosCustomer service | Which source. The customer's belief about which chairs were gifts isn't what the photos show, and one label photo shows the wrong chair. | 56 | 8 | 95% |
| Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer service | Which source. 7 of the 20 asks hinge on records that contradict each other, such as host against guest or feed against register. | 20 | 6 | 100% |
| Tea‑factory ops chat: speak or stay silentSmall business | Leaving alone. Wrong settings are mixed with correct ones, many of which look wrong against the guide but follow the factory's proven practice. | 71 | 6 | 53% |
| Tea factory rooftop solar and battery design with its business caseEnergy engineering | Spotting. The data must be cleaned first: weather in UTC for a site on local time, a month missing from the utility's export. | 18 | 6 | 92% |
| Coffee web‑shop order desk through a four‑act Christmas peakSmall business | Which source. Customers pick the wrong case category about one time in five, and carrier bulletins are late, partial and once false. | 207–213 | 5 | 94% |
| Three‑day hospital ward care coordination, notes‑only memoryHospital | Spotting. Planted errors, such as a dose charted at the wrong amount or an order clashing with an allergy, must each be caught in time. | 290–415 | 6 | 100% |
| Server‑room cooling outage in a 3D building with model‑played staffFacilities | Spotting. Problems nobody mentions must be found, such as a disconnected fire‑suppression cylinder and a contractor's wrong belief about the cable. | 31 | 7 | 33% |
| Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilities | Spotting. The swings come from a hidden open strip behind each new unit, among plausible suspects and small real faults. | 20 | 7 | 50% |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | Leaving alone. The control system's export is out of date in places and writes some tags another way, which aren't discrepancies. | 3,836 | 5 | 100% |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | Leaving alone. The same out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported. | 3,836 | 5 | 99% |
| P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering records | Which source. 44 decisions turn on records that disagree, describe cancelled or temporary work, repeat each other, or claim a change is already drawn. | 65 | 6 | 100% |
| Watchdog program commissioned for a running solvent recovery unitProcess control | Leaving alone. Odd records for equipment that works fine must not be reported as defects, and a one‑point temperature spike must not shut anything down. | 41 | 6 | 93% |
| Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacement | Which source. A rug and a loveseat in the March and July photos aren't what stood at the loss, which only the October photos show. | 25 | 8 | 85% |
| Night repair of a leaking condo floor on an in‑world clockHousehold | Spotting. Two bathroom joints look unsealed but only one leaks, shown only by a coloured‑water test, and the board is wet past the stain. | 13 | 7 | 100% |
| Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solar | Which source. The family's postal number, the register and the photos' location point to different plots, and the plan on file isn't what was built. | 15 | 6 | 99% |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials