Family dinner booking with simulated relatives

The request gives the family's size; the agent finds out the rest

Role

Personal assistant to a family's organiser, working from his laptop.

Request

Every New Year, 40 of the family from seven households eat at his home. This year, hold the dinner at a restaurant for everyone, so that all he has to do is turn up; then tell him what it did, what it decided and why.

Deliverable

The world's end state (a booking confirmed by deposit, the messages sent, the money collected) and a written report.

What the reward depends on

Finding out who is really coming by asking the right relatives; a booking that meets all ten hard constraints; telling one relative privately before the family group; splitting the bill by the family's rule and collecting it into the right account; keeping the organiser's private matters private.

Where the requirements are

In the organiser's files, on the web, and with the relatives and restaurants, who each give a fact only when asked for it. The answer key names the facts, files or logged events behind each scored decision.

18 simulated people hold 136 facts, given only when asked

Interface

A command‑line client in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI or Kimi CLI in our attempts).

Commands

Message or call a contact, post to the family group, wait, search the web, open a page or file, view the bank and make transfers.

World service

One HTTP service holds the phone, the bank, the bookings and the clock, with a separate world for each attempt, and logs every exchange. The grader reads its final state and log. It calls an LLM only to voice the people; code matches questions to facts.

Files

32 files: 7 on the organiser's laptop and 25 web pages (restaurant sites, reviews, maps, 4 menus as PDFs, 6 poster photos), all served through the world service, so every open is time‑stamped.

Web

Web search returns titles and links only; a fact reaches the agent only when it opens the page.

Simulated people

18 voiced by an LLM (9 relatives and 9 restaurants that answer the phone), holding 136 fact cards between them. 5 scripted contacts and 8 restaurants with one recorded line complete the cast.

Voice model

Sonnet 5.5 in our attempts. The service also runs GPT‑5.6 Luna, DeepSeek V4.1 Flash or a model you choose, with the same prompt.

Fact release

Code releases a card when the agent's message asks for it; 4 cards need an earlier card first (a second ask), and some wait on events in the world.

Guards

The model voices only released lines, and its replies are checked for invented names or figures. Catch‑all questions get push‑back and no facts; at most five topics a message are answered.

Same question, same answer

Our board saved replies and replayed them across attempts: the same message after the same exchange got the same reply. In the sample, each container starts with an empty store.

Clock

Starts at 09:00 on a Monday, about three and a half weeks before the dinner. Each message, call, group post or transfer moves it an hour; nights are skipped; reading is free.

Waiting

Waits move the clock by the hours asked. Once asked, each household pays after its own delay.

93 checks, gated on ten hard constraints

Structure

36 result checks (30% of the reward) and 57 decision points (70%): 22 decisions scored as pairs, plus 13 single checks.

Hard constraints

Ten: the date, one room, a seat for everyone, a proper halal arrangement, vegetarian food, the budget per table, distance from home, enough food, step‑free access, and a booking confirmed by deposit.

Gate

Breaking any hard constraint zeroes the 36 result checks and the 44 paired decision checks; only the 13 single decision checks still count.

Right for the right reason

Each paired decision earns one point for the right call, and a second only if the deciding fact was in the attempt's log before the decision, found by the agent's own question, file or bank view. A lucky guess earns half.

Who decides

Scripts decide 88 checks, on the world's state, the log of released facts and the report. Two LLM readers from different model families (Sonnet 5.5 and GPT‑5.6 Luna) decide 5, as stateless yes/no questions; both must agree. Either can run on a model you choose.

Reader agreement

The readers disagreed on 11 of 113 judged results (about 10%, against 3.5% across the office and household environments); each split was settled by written ruling.

Formula

Reward = 0.7 × decision points / 57 + 0.3 × result points / 36, from 0 to 1.

Most attempts broke a hard constraint; none fully solved it

The gate splits the scores: attempts that met all ten hard constraints scored 81–95%, and the rest, keeping only the 13 ungated checks, under 15%. No attempt fully solved it in any version; 26 of 27 earned some reward.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus one line on what to check (each household's numbers, diets and timing, the booking, the money and a record).

Told the rules: here, the request with the hint, plus everything the relatives, restaurants and other contacts would say, and two facts from the files; facts found only in his files are left out.

What the models got wrong

Counted over the 9 models, one attempt each, with only the request, unless a line says otherwise.

1. Hearsay accepted. 7 of 9 attempts never asked a household abroad that another relative had said, second‑hand, wasn't coming; only 2 had a head count from every household before paying the deposit.

2. First answer taken as final. Only 2 attempts asked a hesitant relative a second time and kept her seats; 4 messaged her by a family nickname that isn't a saved contact.

3. Private information disclosed. 3 of 9 attempts told relatives about the organiser's card cashback; across all 27 attempts, none both paid with that card and kept the perk private.

4. Unchecked reports. 6 of 9 final reports gave a head count without the household abroad; 1 flagged that it might have missed someone.

5. Help didn't close the gap. With the hint, 1 of 9 attempts met every hard constraint; told the rules, 4 of 9. Of the five that still failed, four left out the hesitant relative and her husband, whose 'maybe' the request now stated; one stopped within a minute.

What an attempt costs

Length

With only the request, the median attempt took 18 minutes and 59 tool calls (34 to 120 across the 9), and about 1.8 million tokens, cached input included.

Cost

With only the request, the 7 attempts by models with a published price cost $0.48 to $6.23 at list prices (median $2.30). GPT‑6.1 Sol's and GPT‑6 Luna's, priced at an upper bound, were at most $3.31 and $2.71.

What it covers

The agent model only: the simulated people's and readers' calls aren't included. All 27 attempts cost at most $80.

What's free, and what you license

Free sample

Revision 2, with only the request and told the rules, as Harbor tasks with the grader and a reference solution, downloadable after a short form. Free to evaluate in your own stack; training on it needs a licence.

With a licence

Both revisions and all three versions, the world service, the people's fact cards and memories, the grader with its reader prompts, the answer key, and the grader's report on each of the 27 attempts.

Revisions

Revision 2, tested here, adds the organiser's credit card, a cashback offer, a note on his habit, one fact card and three checks to the first.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The sample shares this revision's files, grader and answer key with every lab that requests it, so evaluate on fresh variants. The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• 5 of the 93 checks depend on LLM readers; 11 splits were settled by written ruling.
• The gate concentrates the reward: below it, differences between attempts come from 13 checks.
• The people's voice came from one LLM per attempt. Code releases the facts, but we haven't measured the reward under a second voice model.
• Reading is free, so an attempt that opens every file earns the reason credit for facts in files; facts the people hold still need the right question.
• Doing nothing scores 0 (the grader's self‑test). One regraded copy shows the gate's weight: an attempt told the rules, with its later messages to the family and its money requests removed, still scored 50%.
• An audit on 9 October corrected some grading; its re‑grades aren't on our board yet. They would raise GPT‑6.1 Sol's attempt with only the request from 86% to 88%, and bring Fable 5.1 level with Opus 5.5, the best told the rules, at 95%.
• No red‑team attempts yet.