Clarifying with people
In many jobs, what the agent needs to know sits with people, not in files. Doing well means asking the right person, asking the right question, and weighing what they say.
Three things an agent has to get right
Ask the right person
The deciding facts sit with particular people, such as a parent, a customer or a head baker, and the agent has to find out who.
Ask the right question
People answer only what they're asked, some facts come only on a second ask, and in several, a catch‑all question gets pushed back.
Weigh what they say
Some people misremember, repeat mistakes from their own lists or pass on a rumour, so their answers have to be checked against the records.
They don't ask, ask the wrong thing, or believe what they're told
Not asking the right person
Family dinner booking with simulated relatives: 78% of attempts never asked a household that a relative had heard wasn't coming, and most booked too few seats.
Onboarding bill review: in every attempt, the agent never messaged the client, so the three items only she could explain were never fully settled.
Not asking the right question
12‑month equipment plan: 69% of attempts ordered the new cabinet the manager asked for, and most of those never asked him about a repair.
Seasonal bulk order run with simulated staff: 75% of attempts took the agency's month‑old answer that it was fully booked as final; the attempts that had the rota lead ask again booked a slot that had come free.
Taking answers on trust
Three‑year repair‑or‑replace plan: 92% of attempts ordered the cabinet model the manager already had in mind; only the top attempt chose a different make.
Furniture warranty claim: 58% of attempts accepted a label photo the customer sent for the damaged chair, though it showed another chair.
Partial credit for most attempts, and few full solves
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | People | How it tests this | Checks | Models | Best |
|---|---|---|---|---|---|
| Family dinner booking with simulated relativesHousehold | LLM‑voiced + scripted | Right question. 18 relatives and restaurant staff share facts only for a matching question, some only on a second ask, and push back on catch‑alls. | 93 | 12 | 93% |
| Cancelled‑flight family rebooking against a ticking clockTravel concierge | LLM‑voiced + scripted | Right person. Three family members each hold facts about documents, bookings and who travels with whom, shared only when the right one is asked. | 61–63 | 8 | 85% |
| Onboarding bill review with a simulated clientHousehold bills | LLM‑voiced | Right person. Three items can be settled only by the client, who answers only the right question; the request never mentions messaging her. | 295–320 | 8 | 84% |
| Medical bills review across two health plansHousehold bills | LLM‑voiced | Right question. Two spouses share one message thread and answer from a fixed set of facts, holding what no record shows, within 20 messages. | 287–350 | 8 | 88% |
| Household solar purchase plan with simulated contactsHousehold | LLM‑voiced + scripted | Right person. 11 contacts, from the brother to installers and an insurer, share facts only for a matching question, some only after a follow‑up. | 82 | 10 | 54% |
| Household savings plan from records and familyHousehold | LLM‑voiced + scripted | Right person. Nine people hold deciding figures they give only when asked about them, and the owner himself never answers. | 51 | 8 | 85% |
| Stranded‑family rebooking with minors' exit papers and a dogPersonal assistant | LLM‑voiced | Right person. The facts that decide the route sit with one parent each, released only for the right question, such as the papers' end date. | 34 | 8 | 96% |
| 12‑month equipment plan with simulated shop managerSmall business | LLM‑voiced | Weighing. The manager answers from a partly wrong memory and checks his records only when asked, within ten messages the agent isn't told about. | Cost model | 13 | 57% |
| Seasonal bulk order run with simulated staffSmall business | LLM‑voiced + scripted | Right question. Three staff with partly wrong memories hold key facts, and some come out only on a second ask or after a colleague checks. | 23 | 8 | 75% |
| Three‑year repair‑or‑replace plan with simulated managerSmall business | LLM‑voiced | Right question. The manager answers at most five questions a message, drops the rest, and meets catch‑all requests with a question back. | Cost model | 12 | 93% |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | LLM‑voiced + scripted | Weighing. The manager believes the mistakes in his own list, while the head baker holds the transfers the counts and invoices miss. | 29 | 7 | 99% |
| Gym operations app with three owner‑feedback repair roundsSmall business | LLM‑voiced | Weighing. A simulated owner uses the app after each round and sends short messages; the checks also cover workflows he never touched. | 113 | 7 | 86% |
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Scripted | Right question. The agent's planner can ask each rider at most four quick‑reply questions a request, so each one has to count. | 81 | 5 | 98% |
| Appliance repair triage with budgeted customer questionsCustomer service | Scripted | Weighing. Each question to a scripted customer spends one of a fixed number, and the opening complaint is wrong in 12 of the 80 cases. | 80 | 5 | 100% |
| Furniture warranty claim from a simulated customer's photosCustomer service | LLM‑voiced | Weighing. The chairs' labels exist only in photos the customer sends for a named chair, and each must be matched to the right chair. | 56 | 8 | 95% |
| Server‑room cooling outage in a 3D building with model‑played staffFacilities | LLM‑voiced | Right person. Eight people hold the authority and hands a stopgap needs, from Security to the vendor; four must be engaged within 30 minutes. | 31 | 7 | 33% |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | Scripted | Right question. A scripted desk answers only what is asked, and only it shows which instruments the client's practice says need a control point. | 3,836 | 5 | 100% |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | Scripted | Right question. The same scripted desk answers only what is asked; the client's practice on control‑system points is learnable only from it. | 3,836 | 5 | 99% |
| P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering records | Scripted | Right question. A scripted engineer answers questions charged against a budget, and two items fail unless the one decisive question was asked. | 65 | 6 | 100% |
| Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineering | Scripted | Right question. Six scripted suppliers give stock, delivery and price only when asked, and only asking shows the part that can't arrive in time. | 4 | 5 | 2.0% slower |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials