AsapiThe AI Agent Trials
Asapi / Capabilities / Catching errors
Capability · 40 environments

Catching errors

Records hold mistakes, equipment hides faults, and sources disagree. Doing well means spotting what's wrong, deciding which source to trust, and leaving what's right alone.

What it takes

Three things an agent has to get right

Spot what's wrong

Planted mistakes and faults go unmentioned, such as altered payslips, a wrongly charted dose or a disconnected fire‑suppression cylinder.

Decide which source to trust

Superseded reports, edited records, old invoices and mistaken beliefs contradict the facts, so the agent has to judge which source holds.

Leave what's right alone

Look‑alikes, genuine oddities and correct lines sit among the errors, and in many jobs, flagging them costs reward.

What models get wrong

They miss planted errors, trust the wrong record, or raise false alarms

Errors missed

  1. Bakery wholesale pack with downstream figure checks: 85% of attempts never tested another team's ordering formula on months of different lengths, so the error planted in it went unreported.

  2. Server‑room cooling outage: 43% of attempts handed in their report with a fire‑suppression cylinder still disconnected, a fault shown only on its panel.

The wrong source trusted

  1. Bakery's freezer‑loss insurance claim pack: 79% of attempts used a disposal record that had been edited later, though the same‑day evidence contradicts it.

  2. Kitchen‑fire contents claim: 57% of attempts claimed the worktables and baking trays from old invoices rather than what stood in the kitchen on the night.

False alarms

  1. Temperature compliance register: 58% of attempts handed in a register with invented extra breaches, such as a planned deep clean logged as one.

  2. Watchdog program for a running plant: 33% of attempts cut the heater on a one‑point temperature spike, treating a harmless blip as a fault.

Results

Partial credit for nearly every attempt, and few full solves

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

EnvironmentMedianFully solved
Cancelled‑flight family rebooking against a ticking clock8 models × 3 attempts70%0 of 24Household bill review: card disputes, claims, tracker8 models × 3 attempts86%0 of 24Onboarding bill review with a simulated client8 models × 3 attempts78%0 of 24Medical bills review across two health plans8 models × 3 attempts74%0 of 24Course registration against rules found in portal16 models71%2 of 16Course registration optimised for earliest graduation15 models79%0 of 15Stranded‑family rebooking with minors' exit papers and a dog8 models × 3 attempts61%0 of 24Packaging proof sign‑off for an absent owner13 models2%0 of 13Monthly waste report in house format13 models64%0 of 13Brewery input‑price report in house format12 models73%0 of 12Contractor invoice audit against shop records13 models44%0 of 13Bakery's freezer‑loss insurance claim pack14 models55%0 of 14Bakery wholesale pack with downstream figure checks13 models48%0 of 13Temperature compliance register from messy logger data12 models9%0 of 12Kitchen‑fire contents claim from stale counts, invoices and photos7 models81%0 of 7Bill‑audit and claims app from household files8 models57%0 of 8Refinance‑broker case app from a non‑coder's laptop8 models75%0 of 8Broker app with autonomous lead‑messaging assistant8 models58%0 of 8Listening piano‑practice app: following, transcription, feedback7 models53%0 of 7Gym operations app with three owner‑feedback repair rounds7 models82%0 of 7Offline fault‑finding tablet app for data‑centre cooling units7 models64%0 of 7Transit journey planner and phone telemetry over five field weeks5 models84%0 of 5Returns‑label request hiding unraised account problems8 models × 3 attempts94%0 of 24Retail escalation reply and case records15 models73%0 of 15Missing‑payments reply from error‑laden retail records8 models × 3 attempts78%0 of 24Furniture warranty claim from a simulated customer's photos8 models × 3 attempts82%0 of 24Exact holiday‑rental retrieval over multilingual multi‑source data6 models88%1 of 6Tea‑factory ops chat: speak or stay silent6 models11%0 of 6Tea factory rooftop solar and battery design with its business case6 models66%0 of 6Coffee web‑shop order desk through a four‑act Christmas peak5 models90%0 of 5Three‑day hospital ward care coordination, notes‑only memory6 models95%1 of 6Server‑room cooling outage in a 3D building with model‑played staff7 models5%0 of 7Server‑hall cooling diagnosis by undoable tests in a 3D building7 models35%0 of 7Scanned P&ID set digitised into instrument, line and equipment lists5 models99%1 of 5Photographed P&ID sketches digitised into instrument, line and equipment lists5 models99%0 of 5P&ID as‑built backlog from nine teams' records, drafted in DXF6 models97%1 of 6Watchdog program commissioned for a running solvent recovery unit6 models79%0 of 6Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up8 models × 3 attempts75%0 of 24Night repair of a leaking condo floor on an in‑world clock7 models × 3 attempts41%1 of 21Row‑house rooftop solar designed and quoted from plans, imagery and photos6 models71%0 of 6

Hover a dot for its model and reward.

The 40 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Cancelled‑flight family rebooking against a ticking clockTravel conciergeSpotting. The bookings hold problems the family never raised, such as a schedule change on another leg, or seats lost when a ticket was reissued.61–63885%
Household bill review: card disputes, claims, trackerHousehold billsLeaving alone. Wrong charges hide among legitimate look‑alikes, partial credits and promised refunds, and disputing a look‑alike costs reward.191–262894%
Onboarding bill review with a simulated clientHousehold billsWhich source. Statement names differ from shop names, companies were renamed, and refund messages must be checked against the account.295–320884%
Medical bills review across two health plansHousehold billsSpotting. Problems nobody flagged carry 22% of the weight and must be found record by record, under both health plans' rules.287–350888%
Course registration against rules found in portalUniversity portalSpotting. The starting registration holds problem sections the portal accepts silently, such as a class before the car park opens.1416100%
Course registration optimised for earliest graduationUniversity portalSpotting. Similar starting problems, such as a course without its prerequisite that also meets on work days, have to be found and fixed.141593%
Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantWhich source. The father believes the mother is on the rental contract; she isn't, and only the contract itself shows it.34896%
Packaging proof sign‑off for an absent ownerSmall businessWhich source. A supplier's stated size for a swapped part conflicts with the maker's own sheet, and designs were re‑lettered between rounds.151384%
Monthly waste report in house formatSmall businessLeaving alone. Flagging recoverable data as missing costs reward, as does missing a real gap, a double‑counted till extract or the wrong denominator.431389%
Brewery input‑price report in house formatSmall businessSpotting. A returned load, a double scan beside a genuine look‑alike, a withdrawn surcharge and an unpriced lot all have to be handled.511290%
Contractor invoice audit against shop recordsSmall businessLeaving alone. Planted over- and under‑charges sit among 44 invoice lines, and querying a correct line fails a check.1071361%
Bakery's freezer‑loss insurance claim packSmall businessWhich source. A draft with mistakes, a superseded engineer's report and an edited disposal record conflict with same‑day evidence.601483%
Bakery wholesale pack with downstream figure checksSmall businessSpotting. Planted problems in last month's figures, a customer's rebate and another team's formula must be found, without false alarms.231390%
Temperature compliance register from messy logger dataSmall businessLeaving alone. Genuine unusual entries must be trusted and extra breaches never invented, while faulty sensors, clock errors and a fabricated sheet are caught.1031240%
Kitchen‑fire contents claim from stale counts, invoices and photosSmall businessWhich source. Records true once but not on the night decide five checks, such as a flour count taken before six sacks left.29799%
Bill‑audit and claims app from household filesHouseholdLeaving alone. The app must find wrong charges and route each claim, while pending holds, real repeats and refunds already received are left alone.108861%
Refinance‑broker case app from a non‑coder's laptopSmall businessLeaving alone. Genuine but messy documents mustn't be marked suspect, while those with figures the grader altered must be flagged.51890%
Broker app with autonomous lead‑messaging assistantSmall businessSpotting. As in the refinance‑broker app, statements and payslips with altered or inconsistent figures must be flagged.106868%
Listening piano‑practice app: following, transcription, feedbackHouseholdSpotting. The app's mistake report must list planted wrong, missed, extra and late notes at their bar and beat, and none on a clean take.64758%
Gym operations app with three owner‑feedback repair roundsSmall businessWhich source. Old records are wrong or duplicated: balances must be rebuilt from dated events, and same‑name strangers kept apart.113786%
Offline fault‑finding tablet app for data‑centre cooling unitsFacilitiesSpotting. A blown fuse usually hides a shorted coil or winding, and some causes show only in the alarm history or maintenance log.79787%
Transit journey planner and phone telemetry over five field weeksSoftware for a transit operatorWhich source. Nearly half the graded requests turn on places where the operator's timetable feed is wrong or silent.81598%
Returns‑label request hiding unraised account problemsCustomer serviceSpotting. Problems the customer never raised sit in her account, and a colleague's unsent draft has the label against the wrong return.62898%
Retail escalation reply and case recordsCustomer serviceWhich source. The case file holds misfiled documents, three policy versions and a colleague's draft with wrong figures.114–1151598%
Missing‑payments reply from error‑laden retail recordsCustomer serviceSpotting. Each payment the customer chases turns on a different record error or on a claim the records don't support, and paying twice fails checks.61899%
Furniture warranty claim from a simulated customer's photosCustomer serviceWhich source. The customer's belief about which chairs were gifts isn't what the photos show, and one label photo shows the wrong chair.56895%
Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer serviceWhich source. 7 of the 20 asks hinge on records that contradict each other, such as host against guest or feed against register.206100%
Tea‑factory ops chat: speak or stay silentSmall businessLeaving alone. Wrong settings are mixed with correct ones, many of which look wrong against the guide but follow the factory's proven practice.71653%
Tea factory rooftop solar and battery design with its business caseEnergy engineeringSpotting. The data must be cleaned first: weather in UTC for a site on local time, a month missing from the utility's export.18692%
Coffee web‑shop order desk through a four‑act Christmas peakSmall businessWhich source. Customers pick the wrong case category about one time in five, and carrier bulletins are late, partial and once false.207–213594%
Three‑day hospital ward care coordination, notes‑only memoryHospitalSpotting. Planted errors, such as a dose charted at the wrong amount or an order clashing with an allergy, must each be caught in time.290–4156100%
Server‑room cooling outage in a 3D building with model‑played staffFacilitiesSpotting. Problems nobody mentions must be found, such as a disconnected fire‑suppression cylinder and a contractor's wrong belief about the cable.31733%
Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilitiesSpotting. The swings come from a hidden open strip behind each new unit, among plausible suspects and small real faults.20750%
Scanned P&ID set digitised into instrument, line and equipment listsEngineering recordsLeaving alone. The control system's export is out of date in places and writes some tags another way, which aren't discrepancies.3,8365100%
Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering recordsLeaving alone. The same out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported.3,836599%
P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering recordsWhich source. 44 decisions turn on records that disagree, describe cancelled or temporary work, repeat each other, or claim a change is already drawn.656100%
Watchdog program commissioned for a running solvent recovery unitProcess controlLeaving alone. Odd records for equipment that works fine must not be reported as defects, and a one‑point temperature spike must not shut anything down.41693%
Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacementWhich source. A rug and a loveseat in the March and July photos aren't what stood at the loss, which only the October photos show.25885%
Night repair of a leaking condo floor on an in‑world clockHouseholdSpotting. Two bathroom joints look unsealed but only one leaks, shown only by a coloured‑water test, and the board is wet past the stain.137100%
Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solarWhich source. The family's postal number, the register and the photos' location point to different plots, and the plan on file isn't what was built.15699%

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.