Clarification with simulated users

In these environments, the facts a job depends on sit with simulated relatives, clients, colleagues, customers, riders and building staff, who answer only what they're asked. The agent has to notice what it doesn't know, find the right person and ask.

How the simulated people work

Facts come out only for the right question

Who plays them: An LLM voices the people in 13 of these 17 environments: Sonnet 5.5 in most, and DeepSeek V4.1 Flash at temperature 0 in the 12‑month equipment plan. Code plays the rest: the repair desk's 80 customers, the journey planner's 120 riders and the plant‑drawing jobs' query desks.
Fact cards: In 9 of the 17, each fact is a card that code releases only when the agent's message asks for it; in five, an LLM also matches paraphrases. Some cards need a second ask; some wait on events in the world.
Voice: In those 9, the voice model words only the released lines, and its replies are checked for invented names or figures.
No cards: In the two shop‑manager environments, the manager answers from a short memory, partly wrong by design, and looks up his own records only when asked for a figure, a record or what someone said. Every figure he gives must come from his memory, his records or the conversation.
Asking well: In most, catch‑all questions get push‑back and no facts. Some people answer at most five topics a message; a shop manager leaves after ten messages; repair customers allow a fixed number of questions; each answer at the plant‑drawing desks costs money, and riders drop the app after needless questions.
Wrong by design, and briefs: Some people's memories or briefs are wrong by design, and only the records or the world settle it. In the server‑room incident, eight people play from briefs of what they know, their limits and tasks; in the gym app, the owner answers nothing, reporting up to three concerns a visit.
Same question, same answer: In 15 of the 17, replies are saved and replayed, or come from scripts: an exchange seen before gets the same reply. In the server‑room incident and the gym app, each reply is a fresh model call.
Time: In the household environments, each message moves an in‑world clock by an hour, and a promised reply arrives at the stated time; in the seasonal bulk order, chasing it early costs reward; in the server‑room incident, a phone call costs five world minutes.
Scoring: The reward is on the outcome; in some environments, also on whether the agent asked for the deciding fact before deciding.
Your model as the voice: The people's voice can run on a model you choose: your own or an open‑weight one.
Compared with τ²-bench: There, an LLM plays the user from a scenario listing what it knows. Here, in the fact‑card environments, code releases each fact only to a matching question (an LLM judges paraphrases in five), and the voice words only released lines, so people don't volunteer facts early.

What models get wrong

They take what they're told, and ask too little

Each count covers every model tested on the environment, with only the request; where a model made several attempts, the line says so.

Second‑hand word taken as fact. Family dinner: 7 of 9 attempts never asked a household abroad that another relative said wasn't coming. Only 2 had a head count from every household before paying the deposit.
Asking no one. Household savings plan: 6 of 8 attempts sent no message or call, though only the owner was unavailable. The two that each asked four people were the top two.
Skipping the person it concerns. Household solar purchase: 9 of 10 attempts never contacted the brother whose share they were asked to settle, and 9 of 10 plans got his share wrong.
Not asking twice. Seasonal bulk order: 6 of 8 attempts, the best among them, never booked the agency cover that staff mention only when asked a second time.
The request taken as the plan. 12‑month equipment plan: 8 of 13 attempts ordered only the new cabinet the manager asked for, scoring 0, and used a median of 2 of their 10 messages. Attempts that found what was wrong with the old one and acted scored 44–100%.
The deciding fact not drawn out. Stranded‑family rebooking: in 10 of 24 attempts (8 models × 3) the children flew out after their papers had ended; 5 never asked the mother for the date, and 4 learned it only after choosing the flight.

Results

Full solves in five of the 17

Each dot is one attempt with only the request; the line marks the median. Larger dots are full solves.

EnvironmentMedianFully solved

12‑month equipment plan with simulated shop manager13 models0%1 of 13 Family dinner booking with simulated relatives9 models7%0 of 9 Seasonal bulk order run with simulated staff8 models48%0 of 8 Household solar purchase plan with simulated contacts10 models56%0 of 10 Stranded‑family rebooking with minors' exit papers and a dog8 models × 3 attempts61%0 of 24 Live server‑room cooling incident in a 3D building with model‑played staff7 models65%0 of 7 Three‑year repair‑or‑replace plan with simulated manager12 models68%0 of 12 Cancelled‑flight family rebooking against a ticking clock8 models × 3 attempts70%0 of 24 Household savings plan from records and family8 models73%0 of 8 Medical bills review across two health plans8 models × 3 attempts74%0 of 24 Onboarding bill review with a simulated client8 models × 3 attempts78%0 of 24 Gym operations app with three owner‑feedback repair rounds7 models82%0 of 7 Transit journey planner and phone telemetry over five field weeks5 models84%0 of 5 Kitchen‑fire contents claim from stale counts, invoices and photos7 models86%1 of 7 P&ID as‑built backlog from nine teams' records, drafted in DXF6 models97%1 of 6 Scanned P&ID set digitised into instrument, line and equipment lists5 models99%1 of 5 Appliance repair triage with budgeted customer questions5 models100%3 of 5

Hover a dot for its model and reward.

The 17 environments

How each one releases its facts

Environment	People	How facts are released	Checks	Models	Best, request only	Best, told the rules
12‑month equipment plan with simulated shop managerSmall business (bakery chain; one shop's equipment)	LLM‑voiced	One shop manager (DeepSeek V4.1 Flash at temperature 0; one model's attempts may have had Sonnet 5.5, which the data don't record) answering from a partly wrong memory and his own records; 10 messages an attempt.	Cost model	13	100%	100%
Family dinner booking with simulated relativesHousehold (extended family of seven households)	LLM‑voiced + scripted	18 LLM‑voiced relatives and restaurants; 136 fact cards released by code; second asks; at most 5 topics a message.	93	9	86%	95%
Seasonal bulk order run with simulated staffSmall business (bakery chain supplying a hotel group)	LLM‑voiced + scripted	Three staff voiced by Sonnet 5.5 and 20 scripted contacts; follow‑up facts only on a second ask; replies at the promised time.	23	8	88%	96%
Household solar purchase plan with simulated contactsHousehold (family home; the owner's brother pays the power bill)	LLM‑voiced + scripted	11 LLM‑voiced contacts; fact cards released by code, 6 only on a second ask, one only after another person's card.	82	10	78%	100%
Stranded‑family rebooking with minors' exit papers and a dogA father's personal assistant, working from the family's hub, on the Sunday evening the airline cancels the mother's, two children's and dog's flight home from abroad and moves them to Thursday	LLM‑voiced	The mother and the father	34	8	96%	99%
Live server‑room cooling incident in a 3D building with model‑played staffA police station's facilities duty engineer (FM company), in a 3D model of the station, at the alarm that both live server‑room cooling units have stopped	LLM‑voiced	The duty inspector, the IT service owner, the control room supervisor, the security officer, the FM helpdesk, the level‑4 office manager, the contractor's supervisor…	31	7	77%	100%
Three‑year repair‑or‑replace plan with simulated managerSmall business (bakery chain; one shop's equipment)	LLM‑voiced	The same manager (Sonnet 5.5): at most 5 questions answered a message; 10 messages an attempt.	Cost model	12	99%	99%
Cancelled‑flight family rebooking against a ticking clockA travel concierge's trip desk on the evening a family's overnight flight to a wedding abroad is cancelled	LLM‑voiced + scripted	Three family members voiced by Sonnet 5.5; 17 fact cards matched by code or an LLM, some only on a second ask; an airline chat agent voiced by Sonnet 5.5, with code deciding its offers.	61–63	8	85%	99%
Household savings plan from records and familyHousehold (multi‑generation family home)	LLM‑voiced + scripted	9 LLM‑voiced contacts; 80 fact cards, released by code or an LLM's yes/no judgement; 5 only on a second ask.	51	8	89%	98%
Medical bills review across two health plansThe same firm's review desk; a couple's medical and pharmacy bills for a plan year, with two employer plans, a pharmacy benefit, a flexible spending account and a card	LLM‑voiced	Two spouses in one thread; 23 fact cards matched by code, with an LLM for paraphrases; 20 messages an attempt.	287–350	8	88%	99%
Onboarding bill review with a simulated clientThe same bill‑management firm's review desk; a new client's first review, with a client message thread	LLM‑voiced	One client; 25 fact cards matched by code, with an LLM classifier for paraphrases; 20 messages an attempt.	295–320	8	84%	95%
Gym operations app with three owner‑feedback repair roundsAn independent gym in an office tower: the owner coaches 22 people, with three freelance trainers, a front desk and a staff contract just signed with the bank upstairs	LLM‑voiced	Nobody answers during the build	113	7	86%	86%
Transit journey planner and phone telemetry over five field weeksEngineering of a public‑transport journey‑planning app for a made‑up city's metro, tram, bus, express‑bus and ferry network, used between releases by 120 simulated riders	Scripted	Each rider has a home, two to four regular places, a schedule, a way of carrying the phone, a mode habit and a comfort sensitivity (some need step‑free access or carry…	81	5	98%	–
Kitchen‑fire contents claim from stale counts, invoices and photosSmall business: a bakery chain, one shop's back kitchen the morning after a fire	LLM‑voiced + scripted	The shop manager, the head baker and the insurer's loss adjuster answer live from 47 fact cards (16, 13 and 18: 35 records, 7 beliefs, all the manager's, and 5…	29	7	100%	100%
P&ID as‑built backlog from nine teams' records, drafted in DXFDrawing office for a made‑up process plant unit (two trains, cooling water, a flare knock‑out drum) whose P&IDs, held as a nine‑sheet DXF CAD master, have fallen six months behind the plant; the agent is a contract drafter	Scripted	The desk's code answers ask ITEM QUESTION from a fixed table per backlog item, for one of six questions (which equipment, valve type, location, permanent, same job,…	65	6	100%	–
Scanned P&ID set digitised into instrument, line and equipment listsEngineering records for a made‑up process plant unit (two process trains, cooling water, a flare knock‑out drum) with no CAD source; the agent is a digitisation consultant on a fixed price	Scripted	The desk's fixed code answers each query from the client's as‑built record: a tag, line or equipment check names the first field of the agent's current row that differs…	3,836	5	100%	–
Appliance repair triage with budgeted customer questionsAppliance repair business's customer desk, earning only on parts and labour	Scripted	80 scripted customers; each question spends one of a case's 6 to 25; the opening complaint is wrong in 12 of 80.	80	5	100%	–

Cost model: the reward prices the agent's plan, from a baseline plan (0) to the best a search found (1). Told the rules: both equipment plans give the manager's records, and the family dinner what the people would say. A range: the count varies by attempt. A dash: not tested.

Known limitations

What we haven't measured

In our attempts, each environment's people ran on one LLM, as far as our records show; in the 12‑month equipment plan, one model's attempts may have had Sonnet 5.5. We haven't measured how the reward changes under another.
An agent in training could learn the words that release facts. Catch‑all questions release nothing and messages are capped, but nobody has tried to game this on purpose yet.
Simulated people are consistent by construction; real people are less so.

Try a sample environment

Three are downloadable as Harbor tasks, with graders and reference solutions. Request one with your work email, or book a call about the other 53.

Request a sample Book a 30‑minute call