Family dinner booking with simulated relatives
ID house‑rules/task11 · revision 2 · results as of 17 November 2026
From the organiser's laptop, the agent books a restaurant for a family dinner of about 40 people from seven households, tells every household what it needs to know, collects each share and reports back.
One of our three samples: Harbor tasks with their own grader and a reference solution, free to download after a short form (1.8 MB).
- Checks
- 93, 10 of them hard constraints
- Tested
- 12 models, 3 attempts each
- Best attempt
- 93% (Fable 5.2)
- Median
- 10%
- Full solves
- 0 of 36
- Median attempt
- 18 min
A dinner for 40 from seven households, booked so the organiser only has to turn up
Clarifying with people
Relatives and restaurants share a fact only for a matching question, some only on a second ask, and push back on catch‑alls.
Stateful tool use
Holds expire, payments land hours later, and one confirmation carries an error until queried, so every booking and payment needs checking.
Disclosure and honesty
Telling a relative about the organiser's private card perk costs checks, as does telling the family group before one relative hears privately.
When to act or escalate
An unpaid household gets one private, gentle reminder, only once the bank shows nothing; a second within 72 hours fails that check.
A world service with a phone, a bank, bookings and a clock
The agent's world
A command‑line client in a container, through the model's usual coding agent, to message, call, search, open pages and use the bank.
The people
18 relatives and restaurants voiced by an AI model from 136 fact cards; code decides which fact a question releases.
The clock
Each message, call or transfer moves it an hour, nights are skipped and reading is free; the dinner is three and a half weeks away.
The reward
Breaking any of ten hard constraints zeroes most checks, and a right decision earns full credit only if its deciding fact was found first.
17% of attempts met every hard constraint and scored 82–93%; the rest scored 12% or less
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| GPT‑6 AstraOpenAI | 63% |
| Fable 5.2Anthropic | 62% |
| GPT‑6.1 SolOpenAI | 61% |
| Kimi K3Moonshot | 10% |
| DeepSeek V4.1 FlashDeepSeek | 10% |
| Sonnet 5Anthropic | 10% |
| GLM‑5.3‑FlashZhipu | 9% |
| Fable 5.1Anthropic | 9% |
| Opus 5.5Anthropic | 8% |
| Opus 5Anthropic | 7% |
| Sonnet 5.5Anthropic | 7% |
| GPT‑6 LunaOpenAI | 2% |
With every fact the relatives and restaurants could give added to the request, 75% of the 36 attempts met every hard constraint.
They take hearsay, pay before the head count, and let private details out
Clarifying with people
78% of attempts never contacted a household abroad that another relative had said, second‑hand, wasn't coming.
Only 19% had a head count from every household before paying the deposit.
Disclosure and honesty
28% of attempts told relatives about the organiser's private card cashback, including five of the six that met every hard constraint.
When to act or escalate
All six that met every hard constraint reminded the household that believed it had paid, and in five a wording check found the reminder implied it hadn't.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 2.4 million tokens, cached input included.
What's in it
The world service, the people's fact cards, the request in all three versions, the full grader with a reference solution, and per‑check results.
The sample
Harbor tasks in two of the three versions, with their own grader and a reference solution, free for evaluation.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
Its files and grader go out as a sample, so hold out fresh variants for clean evaluation; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials