When to act or escalate
Most jobs have moments to act, to wait and to call someone. Doing that well means acting in time, holding back from needless action, and raising the alarm with the right person.
Three things an agent has to get right
Act when it's needed
Some actions can't wait: pausing a booked clearance, closing a valve on a leak, stopping a leaking pump.
Hold back when it isn't
Restraint checks fail needless action, such as unneeded letters, repeat reminders, alarms over a harmless blip, or questions and checks spent for nothing.
Raise the alarm with the right person
Some problems belong to someone else, such as the right physician, a manager or a vendor, and the agent has to reach them in time.
They act too late, do too much, or don't raise the alarm
Too little, too late
Cancelled‑flight family rebooking: in 58% of attempts, the no‑cost seats sold out before the agent booked them.
Retail escalation reply and case records: a refund the card issuer had sent back was due again that day, and 67% of attempts didn't pay it.
Too much
Tea‑factory ops chat: in 83% of attempts, the agent told the crew to change course at some of the 16 points where the crew was right; the worst did so 15 times.
Tea‑factory cash desk: 60% of attempts asked the owner for money in scenarios that could get through without it.
Not raising the alarm
Three‑day hospital ward care coordination: 50% of attempts didn't page a doctor in time about a dose charted at the wrong amount.
Watchdog program for a running plant: 33% of attempts treated a report of a leak that couldn't be shut off as information, and never raised the alarm.
Partial credit for nearly every attempt, and few full solves
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Family dinner booking with simulated relativesHousehold | Holding back. An unpaid household gets one private, gentle reminder, only once the bank shows nothing; a second within 72 hours fails. | 93 | 12 | 93% |
| Cancelled‑flight family rebooking against a ticking clockTravel concierge | Acting. Paying while a free change exists fails the money checks, yet waiting to ask or verify can lose the free seats. | 61–63 | 8 | 85% |
| Household bill review: card disputes, claims, trackerHousehold bills | Holding back. Each charge must be routed, from a card dispute to leaving it alone, and needless letters fail as surely as missed ones. | 191–262 | 8 | 94% |
| Onboarding bill review with a simulated clientHousehold bills | Holding back. Disputing items the client could have explained, or writing letters where none is due, fails; promised refunds are held as pending. | 295–320 | 8 | 84% |
| Medical bills review across two health plansHousehold bills | Holding back. The firm writes once to a plan administrator rather than to providers whose bills will drop, and holds bills while a plan reprocesses. | 287–350 | 8 | 88% |
| Household solar purchase plan with simulated contactsHousehold | Escalating. Checks reward reaching the right people, from the owner's brother to the power company, while pestering family or friends costs reward. | 82 | 10 | 54% |
| Stranded‑family rebooking with minors' exit papers and a dogPersonal assistant | Holding back. Buying flights while a free rebooking exists fails, while the second driver and the airline's statement must be secured that night. | 34 | 8 | 96% |
| Seasonal bulk order run with simulated staffSmall business | Holding back. Chasing a colleague before the time they promised costs reward, yet leaving a staff request unanswered fails. | 23 | 8 | 75% |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | Acting. The booked clearance must be held by a message before Wednesday 08:00, while nothing claim‑like may reach the adjuster before her review. | 29 | 7 | 99% |
| Broker app with autonomous lead‑messaging assistantSmall business | Acting. The app's assistant must answer new leads within minutes and chase silent ones, but never double‑message or promise a rate. | 106 | 8 | 68% |
| Unattended train‑ticket buyer for a rate‑limited booking siteHousehold | Acting. Returned seats last under two minutes, so checking too seldom loses them, while going over the site's limits pauses the account. | 26 | 8 | 100% |
| Smart‑home retrofit: purchase plan, wiring and local control softwareHousehold | Holding back. The house shuts the water on a leak and windows in steady rain, yet ignores rain glitches and keeps the owner's override. | 121 | 7 | 92% |
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Holding back. From a rider's fifth trip on a journey, extra questions annoy, and three annoyances lose the rider, yet too few leave needs unknown. | 81 | 5 | 98% |
| Returns‑label request hiding unraised account problemsCustomer service | Acting. 20% of the reward is for acting today on problems the customer never raised, while goodwill she isn't owed fails. | 62 | 8 | 98% |
| Appliance repair triage with budgeted customer questionsCustomer service | Holding back. Each next step is graded exactly, so sending a technician where the customer can fix it fails as surely as doing too little. | 80 | 5 | 100% |
| Retail escalation reply and case recordsCustomer service | Escalating. Each amount must be paid today, referred to the manager, routed or declined, and paying what must be referred fails. | 114–115 | 15 | 98% |
| Month‑long parcel fleet dispatch, notes‑only memorySmall business | Holding back. Wake‑ups are capped at four a day, so a watcher that fires on known or harmless events leaves later breakdowns unanswered. | 29 | 4 | 100% |
| Tea‑factory cash desk over thirteen weekly sessionsSmall business | Holding back. Asking the owner for money fails when the business could manage without it, and asking too late lets it go under. | 64 | 5 | 100% |
| Coffee roaster materials planning across two seasonsManufacturing | Holding back. Preparing for the manager's real tip is rewarded, while buying on a tip that comes to nothing or a withdrawn warning fails. | 112–166 | 5 | 98% |
| Tea‑factory ops chat: speak or stay silentSmall business | Holding back. 35 check points need the agent to speak up and 22 to stay silent, and wrong interruptions are taken off the catches. | 71 | 6 | 53% |
| Coffee web‑shop order desk through a four‑act Christmas peakSmall business | Holding back. Restraint checks fail action the records don't support, such as moving a region on a bulletin alone. | 207–213 | 5 | 94% |
| Three‑day hospital ward care coordination, notes‑only memoryHospital | Escalating. Each page the manual calls for counts, and each page it doesn't, or one sent to the wrong physician, counts against the agent. | 290–415 | 6 | 100% |
| Server‑room cooling outage in a 3D building with model‑played staffFacilities | Escalating. Actions must be timed and within each person's authority: the works stopped early, and the fire door held open only through Security. | 31 | 7 | 33% |
| Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilities | Holding back. The agent is there to find out, not repair: cooling kept, everything put back, yet the cause proven by a change it undoes. | 20 | 7 | 50% |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | Holding back. Desk checks cost money against a fixed price, so checking every row is unaffordable, yet a guessed digit fails a critical check. | 3,836 | 5 | 100% |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | Holding back. The same priced desk: checking everything costs more than the job pays, and a guessed digit nothing supports fails. | 3,836 | 5 | 99% |
| P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering records | Escalating. Holding an item the records settle is wrong, as is drawing a guess; one change can't be placed at all and must be held. | 65 | 6 | 100% |
| Watchdog program commissioned for a running solvent recovery unitProcess control | Acting. The watchdog must stop the right pump in a leak and escalate only inside the leak window, without shutting down on a harmless afternoon. | 41 | 6 | 93% |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials