Stranded‑family rebooking with minors' exit papers and a dog

The request asks for a cheap way home; the facts that decide it sit with one parent each

Role

The father's personal assistant, working only through the family's hub and nine sites with delegated logins.

Request

The father writes that the airline cancelled the flight home and moved the family to Thursday; they must be home by Wednesday night, cheaply, with anything else the change affects sorted. The assistant may book, change, cancel and pay without asking, and message both parents.

Deliverable

The world's end state (bookings, refunds, purchases, the rental, pet bookings, claims, messages), a note for the parents of at most 600 words and a money‑plan CSV.

What the reward depends on

A legal route home by Wednesday, though the children's exit papers expire before the airline's own rebooking; the free rebooking used rather than new tickets; the mother added as a driver; the dog left as agreed; refunds and claims on the right cover; the note's content and four formatting conventions.

Where the requirements are

In two mailboxes, 14 family documents, the assistant's four past trip summaries and the sites' own rules, and with the two parents, who each hold facts no page shows. No page states the conventions; the four past summaries show them.

A family hub, nine sites and two LLM‑voiced parents, on a frozen clock

Interface

A text browser over MCP with nine tools (open, click, read, scroll, back, type, submit, a calculator and a word counter), driven by Claude Code or Codex CLI on a Mac. No shell and no web.

Isolation

No container. Claude Code's file tools were confined by a macOS profile to its own folder, with Bash and web tools off; Codex CLI ran read‑only, web search off, with a file tool limited to the output folder.

The hub

Two mailboxes (97 and 88 messages), message threads with each parent and the family group chat, 14 family documents in three folders and the assistant's four past trip summaries, all as text pages.

Sites

The airline (changes, refunds, a written statement, pets, passenger‑rights pages), a flight search, a second airline's seats, the rental car, a licensed pet shipper, the travel insurer, the card's benefits portal, a document‑rules lookup and a dog import form. None has a chat agent.

Simulated people

The mother and the father, voiced by Sonnet 5.5 from 33 fact cards (23 and 10); the grandmother in the group chat never replies. At most 20 messages a person, unannounced, and 5 topics a message.

Fact release

Code releases cards by the content of a message, with one cached LLM classifier call where code releases nothing. A reply that adds, drops or changes a fact is sent back once, then replaced by the cards' text. Every message was audited blind.

Clock

Frozen on the Sunday evening: no action moves it and no hold lapses.

Limits

One session of up to 150 minutes and 1,000 turns, with a $60 cap at API prices for Claude Code.

34 checks in five families, weighted by layer; a failed exit is charged once

Structure

34 checks in five families: the children's exit and the way home (8), the car (7), the dog (7), money and claims (8) and how the assistant writes (4). Layers weigh 35 legal and on time, 25 net cost, 15 the family's requests, 15 the note's content and 10 style.

One cause, one charge

Each loss is charged to its root cause: the children's exit and no route home are capped at 55 points, every other cause at 15. The five families group these causes, so a family can lose more.

Right for the right reason

The exit check passed without the paper's end date ever released, or the dog check without the mother's agreement drawn out before the note's last version, earns a third of its weight.

Who decides

A script reads the final state, the action log and both files. Seven of the 13 note items, 8.25 points, go to two blind Opus 5.5 readers when the script can't settle them, with known‑verdict references in every batch.

Reader agreement

In the 24 attempts with only the request, the readers decided 75 note‑item results; each of 2 splits was settled by a written ruling, logged with its reason, and a batch that misses a reference is rejected.

Formula

Reward = 1 minus the sum of capped losses, each a check's weight times its missing credit. Each formatting convention is credited as the share of its instances that follow it.

The exit papers split the field, the house style cost every attempt, and none fully solved it

With only the request, 12 of 24 attempts got the children out of the country while their papers still allowed it, having asked the mother for the date; 10 flew them after the papers ended. Told the rules, both models tested scored 99%.

Each dot is one attempt; the line marks the median.

With only the request, each model made three attempts, and the table shows their mean. Told the rules, two models made one attempt each.

Told the rules: the request plus one paragraph stating every rule, every formatting convention and whom to ask, and to ask again after a plea or a reassurance; it gives no answers.

With only the request, the spread between a model's three attempts ran from 0 points (GPT‑6 Sol) to 56 (GPT‑6.1 Sol).

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Exit papers missed. In 10 of 24 attempts the children flew out after their papers had ended: 5 never drew the date from the mother, 4 learned it after choosing the flight and kept it, and 1 moved them to a later flight counting on a new paper.

2. The household's formats missed. All 23 notes broke at least one of the four formatting conventions every past summary follows (the 24th attempt wrote none): 1 of 55 to‑do lines, 34 of 176 times and 27 of 88 bookings followed them.

3. Claims without an owner. 18 of 24 attempts missed the claims item, 8 of them among the 13 that got the family home legally; at least 9 of the failing notes gave the deadline but named no one to file or take the next step.

4. Stopping at once. 2 of 24 attempts ended within a minute: one changed nothing and saved no file, the other made a single booking change.

What an attempt costs

Length

With only the request, the median attempt took 9 minutes and about 209 tool calls (21 to 303 across the 24).

Cost

Recorded for 12 of the 24 attempts with only the request, the four Anthropic models', at list prices by Claude Code's own count; the OpenAI models' agent logs tokens but not cost. With half unrecorded, we give no median or range.

Not included

Cost covers the agent model only: the model calls that voice the parents, the classifier, the readers and the audit aren't included.

What you license

With a licence

Revision 1 in both versions: the hub and its sites, the parents' service with its fact cards, the grader with its reader instructions and reference notes, the answer key, a reference solution, and the grader's report on each of the 26 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today it runs on a Mac in a macOS sandbox, not a container, with standard‑library Python for the hub, the parents' service and the grader; the voices and readers are reached through the Claude Code CLI.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Its job page on The AI Agent Trials, our public benchmark, describes the job, its main pitfalls and each model's results in plain language, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Doing nothing scores about 30% in the grader's self‑test: it accepts the airline's Thursday flight and writes a short note and money plan in the house style. An attempt that changed and saved nothing scored 14%.
• Taking a legal early flight without asking about the papers scores about 90%: the exit check then earns a third. Luck on the route is discounted, not ruled out.
• The clock is frozen, so seats never sell out and holds never lapse: racing the clock isn't tested here, unlike the cancelled‑trip environment.
• The formatting conventions are 10% of the reward, and a note with no instance of one earns nothing for it.
• Told the rules was tested on two models, once each, which can't separate the stated rules from the variation between attempts.
• Eight attempts were made again after the blind audit found the parents' service had withheld or wrongly released a deciding fact; 81 harmless release misses in the counted attempts are kept as notes.