AsapiThe AI Agent Trials
Asapi / Capabilities / Clarifying with people
Capability · 20 environments

Clarifying with people

In many jobs, what the agent needs to know sits with people, not in files. Doing well means asking the right person, asking the right question, and weighing what they say.

What it takes

Three things an agent has to get right

Ask the right person

The deciding facts sit with particular people, such as a parent, a customer or a head baker, and the agent has to find out who.

Ask the right question

People answer only what they're asked, some facts come only on a second ask, and in several, a catch‑all question gets pushed back.

Weigh what they say

Some people misremember, repeat mistakes from their own lists or pass on a rumour, so their answers have to be checked against the records.

What models get wrong

They don't ask, ask the wrong thing, or believe what they're told

Not asking the right person

  1. Family dinner booking with simulated relatives: 78% of attempts never asked a household that a relative had heard wasn't coming, and most booked too few seats.

  2. Onboarding bill review: in every attempt, the agent never messaged the client, so the three items only she could explain were never fully settled.

Not asking the right question

  1. 12‑month equipment plan: 69% of attempts ordered the new cabinet the manager asked for, and most of those never asked him about a repair.

  2. Seasonal bulk order run with simulated staff: 75% of attempts took the agency's month‑old answer that it was fully booked as final; the attempts that had the rota lead ask again booked a slot that had come free.

Taking answers on trust

  1. Three‑year repair‑or‑replace plan: 92% of attempts ordered the cabinet model the manager already had in mind; only the top attempt chose a different make.

  2. Furniture warranty claim: 58% of attempts accepted a label photo the customer sent for the damaged chair, though it showed another chair.

Results

Partial credit for most attempts, and few full solves

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

EnvironmentMedianFully solved
Family dinner booking with simulated relatives12 models × 3 attempts10%0 of 36Cancelled‑flight family rebooking against a ticking clock8 models × 3 attempts70%0 of 24Onboarding bill review with a simulated client8 models × 3 attempts78%0 of 24Medical bills review across two health plans8 models × 3 attempts74%0 of 24Household solar purchase plan with simulated contacts10 models24%0 of 10Household savings plan from records and family8 models50%0 of 8Stranded‑family rebooking with minors' exit papers and a dog8 models × 3 attempts61%0 of 2412‑month equipment plan with simulated shop manager13 models0%0 of 13Seasonal bulk order run with simulated staff8 models46%0 of 8Three‑year repair‑or‑replace plan with simulated manager12 models0%0 of 12Kitchen‑fire contents claim from stale counts, invoices and photos7 models81%0 of 7Gym operations app with three owner‑feedback repair rounds7 models82%0 of 7Transit journey planner and phone telemetry over five field weeks5 models84%0 of 5Appliance repair triage with budgeted customer questions5 models100%3 of 5Furniture warranty claim from a simulated customer's photos8 models × 3 attempts82%0 of 24Server‑room cooling outage in a 3D building with model‑played staff7 models5%0 of 7Scanned P&ID set digitised into instrument, line and equipment lists5 models99%1 of 5Photographed P&ID sketches digitised into instrument, line and equipment lists5 models99%0 of 5P&ID as‑built backlog from nine teams' records, drafted in DXF6 models97%1 of 6Club race‑car design and parts procurement on a web app with hidden supplier terms5 models96%0 of 5

Hover a dot for its model and reward.

The 20 environments

How each environment tests it

EnvironmentPeopleHow it tests thisChecksModelsBest
Family dinner booking with simulated relativesHouseholdLLM‑voiced + scriptedRight question. 18 relatives and restaurant staff share facts only for a matching question, some only on a second ask, and push back on catch‑alls.931293%
Cancelled‑flight family rebooking against a ticking clockTravel conciergeLLM‑voiced + scriptedRight person. Three family members each hold facts about documents, bookings and who travels with whom, shared only when the right one is asked.61–63885%
Onboarding bill review with a simulated clientHousehold billsLLM‑voicedRight person. Three items can be settled only by the client, who answers only the right question; the request never mentions messaging her.295–320884%
Medical bills review across two health plansHousehold billsLLM‑voicedRight question. Two spouses share one message thread and answer from a fixed set of facts, holding what no record shows, within 20 messages.287–350888%
Household solar purchase plan with simulated contactsHouseholdLLM‑voiced + scriptedRight person. 11 contacts, from the brother to installers and an insurer, share facts only for a matching question, some only after a follow‑up.821054%
Household savings plan from records and familyHouseholdLLM‑voiced + scriptedRight person. Nine people hold deciding figures they give only when asked about them, and the owner himself never answers.51885%
Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantLLM‑voicedRight person. The facts that decide the route sit with one parent each, released only for the right question, such as the papers' end date.34896%
12‑month equipment plan with simulated shop managerSmall businessLLM‑voicedWeighing. The manager answers from a partly wrong memory and checks his records only when asked, within ten messages the agent isn't told about.Cost model1357%
Seasonal bulk order run with simulated staffSmall businessLLM‑voiced + scriptedRight question. Three staff with partly wrong memories hold key facts, and some come out only on a second ask or after a colleague checks.23875%
Three‑year repair‑or‑replace plan with simulated managerSmall businessLLM‑voicedRight question. The manager answers at most five questions a message, drops the rest, and meets catch‑all requests with a question back.Cost model1293%
Kitchen‑fire contents claim from stale counts, invoices and photosSmall businessLLM‑voiced + scriptedWeighing. The manager believes the mistakes in his own list, while the head baker holds the transfers the counts and invoices miss.29799%
Gym operations app with three owner‑feedback repair roundsSmall businessLLM‑voicedWeighing. A simulated owner uses the app after each round and sends short messages; the checks also cover workflows he never touched.113786%
Transit journey planner and phone telemetry over five field weeksSoftware for a transit operatorScriptedRight question. The agent's planner can ask each rider at most four quick‑reply questions a request, so each one has to count.81598%
Appliance repair triage with budgeted customer questionsCustomer serviceScriptedWeighing. Each question to a scripted customer spends one of a fixed number, and the opening complaint is wrong in 12 of the 80 cases.805100%
Furniture warranty claim from a simulated customer's photosCustomer serviceLLM‑voicedWeighing. The chairs' labels exist only in photos the customer sends for a named chair, and each must be matched to the right chair.56895%
Server‑room cooling outage in a 3D building with model‑played staffFacilitiesLLM‑voicedRight person. Eight people hold the authority and hands a stopgap needs, from Security to the vendor; four must be engaged within 30 minutes.31733%
Scanned P&ID set digitised into instrument, line and equipment listsEngineering recordsScriptedRight question. A scripted desk answers only what is asked, and only it shows which instruments the client's practice says need a control point.3,8365100%
Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering recordsScriptedRight question. The same scripted desk answers only what is asked; the client's practice on control‑system points is learnable only from it.3,836599%
P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering recordsScriptedRight question. A scripted engineer answers questions charged against a budget, and two items fail unless the one decisive question was asked.656100%
Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineeringScriptedRight question. Six scripted suppliers give stock, delivery and price only when asked, and only asking shows the part that can't arrive in time.452.0% slower

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.