Cancelled‑flight family rebooking against a ticking clock

The request asks for a rescue; each traveller holds facts no page shows

Role

Trip‑desk assistant at a travel concierge, working only through the firm's desk system.

Request

The client writes that the family's overnight flight is cancelled and the airline has moved them around. All four must reach a wedding's family photo, together if possible; the desk may spend up to a set budget without asking, and contact the family, airline and hotel.

Deliverable

The world's end state (bookings, refunds, purchases, seats, assistance, messages), an itinerary as CSV and a note to the family of at most 700 words.

What the reward depends on

Everyone there in time on routes their documents allow; free seats secured before they sell out, with no needless purchase or refund; facts asked of the traveller who holds them, twice where needed; the rest of the trip checked; a note written the way the firm's past notes are.

Where the requirements are

In the client file, the family's mailbox, the firm's playbook and four past trip notes, the sites' own rules, and with three family members who each hold facts no page shows. No page states the firm's note habits.

Three simulated travellers, an airline agent and seats that sell on a clock

Interface

A text browser over MCP with nine tools (open, click, read, scroll, back, type, submit, a calculator and a word counter), driven by Claude Code or Codex CLI on a Mac. No shell and no web.

Isolation

No container: each agent ran in a macOS sandbox with no shell. The Claude models' profile denied the task folders, answer key, fact cards and grader; the GPT models' only file tool reached the two output files. Probes checked each sandbox before every attempt.

The desk

One local web app renders the firm's desk and every site, with a separate state for each attempt. Every booking change, refund, purchase, seat, assistance request and message is logged, so an attempt can be replayed.

Sites

The airline's booking pages and chat agent, eight other airlines' booking pages, a fare‑search site that sells seats on the card on file, a travel‑document rules lookup and the hotel's platform. The timetable holds 81 flights on nine airlines.

Records

The client file (seven clients, the family's profiles and five document scans as text), a mailbox of 78 messages, most of them everyday mail, and a wiki of 11 pages, among them the playbook and four past trip‑change notes.

Simulated people

Three family members voiced by an LLM from 17 fact cards, each answering at most 20 messages. The family's group chat and the hotel give fixed lines.

Fact release

Code releases cards by matching the message, with an LLM classifier for paraphrases on 13 of them; 2 come out only on a second ask and 2 only to a follow‑up. A reply that adds or changes a fact is replaced by the cards' text.

Airline agent

Code decides every offer, answer and action. An LLM reads each chat message into a structured request and words the answer from a fixed fact sheet, which a verifier checks; an offer is carried out only after an explicit yes.

Voice model

Sonnet 5.5 for the family's voices and the airline agent's reading and wording in our attempts, through the Claude Code CLI; a model you choose on request.

Clock

Starts on the cancellation evening and moves only with the agent's requests: about 30 seconds a page or form, 2 minutes an airline chat message, 3 to 6 minutes before a family reply. Reading, typing and writing files are free.

Seats

Other passengers take seats on the alternative flights at fixed in‑world times, the same in every attempt. Changes through the airline's agent are free and reversible, but seats it gives up are held only briefly.

Limits

One session of up to 150 minutes and 1,000 turns.

61 checks in nine weighted layers; the firm's note habits carry 30%

Structure

61 checks (63 in an attempt, with the note's group items one per group), in nine layers that sum to 100: trip state 24, whole trip 9, time 4, money 6, hotel 3, itinerary 7, note 14, house style 30, what stays out 3.

House style

Five habits shown only in the firm's past notes, weighted 3, 3, 10.5, 10.5 and 3. Each scores the share of its instances in the note that follow it.

One cause, one charge

Each check carries the family of its root cause; a check that fails because of an earlier miss joins that miss's family, and each such family costs at most 8 points.

Right for the right reason

Two checks, on a traveller's documents and her companion, give a third of their worth when the outcome is right but the deciding answer from the family was never read. A lucky guess earns a third.

Who decides

Scripts decide the checks on the world's state, the action log, the clock and the files: about 87 of 100 points. Two blind LLM readers, both Opus 5.5, read the 15 note items a script can't settle, with known‑verdict reference notes in every batch.

Reader agreement

Both readers matched all 210 known verdicts and split on 8 of 356 items; each split was settled by a written ruling, logged with its reason: all 8 passed. After the audit, 7 agreed fails were re‑read as passes; 15 results carry a ruling.

Chat audit

An independent blind auditor (Opus 5.5) read every airline chat for engine faults. Faulted attempts were made again once; 6 of the 24 attempts with only the request still met a confirmed fault and count, marked as faulted.

Formula

Reward = the weighted share of applicable checks passed, from 0 to 1. An attempt with no action, message or file passes nothing.

Seats lost to the clock and the note's format decided the order; none fully solved it

With only the request, every model but one got all four travellers there on valid routes in every attempt, keeping the hotel; seats lost while checking and the note's house style decided the order. Told the rules, the two models tested scored 96% and 99%.

Each dot is one attempt; the line marks the median.

With only the request, each model made three attempts, and the table shows their mean. Told the rules, two models made one attempt each.

Told the rules: the request plus one paragraph stating every travel rule, the clock's costs, all five house habits, whom to ask about what (a text the airline may have sent the booking contact among them) and when to ask again; it gives no answers.

With only the request, the spread between a model's three attempts ran from 1 point (GPT‑6 Astra) to 23 (GPT‑6 Luna); Opus 5.5's scored 76%, 83% and 85%.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Seats lost while checking. In 14 of 24 attempts the free seats for the child and a parent sold out while the agent was still asking or checking; all 14 then bought other tickets, $680 to $1,070 after refunds. GPT‑6 Sol secured them first every time.

2. The firm's habits missed. No attempt wrote every to‑do line or every amount the way all four past notes do: 6 of 67 to‑do lines and 33 of 103 amounts matched. Told the rules, Opus 5.5 matched every one.

3. The family's choice not followed. Only 10 of the 21 attempts that rebooked the family sent the grandmother with the companion the family chose. The mother's first reply is a plea for all four together; the choice comes only on a second ask.

4. Documents taken on trust. 9 of the 21 attempts that rebooked the family never read the grandmother's own answer on her documents; her route happened to suit them each time, earning a third of the credit.

5. Stopping at once. GPT‑6 Luna ended all 3 of its attempts within a minute, without changing a booking or messaging anyone.

What an attempt costs

Length

With only the request, the median attempt took 10 minutes and 167 tool calls (14 to 251 across the 24), and about 10 million tokens, cached input included.

Cost

Recorded for 12 of the 24 attempts with only the request, the four Anthropic models', at list prices by Claude Code's own count; the OpenAI models' agent logs tokens but not cost. With half unrecorded, we give no median or range.

Not included

Cost covers the agent model only: the family's voices, the airline agent's model calls, the readers and the chat audit aren't included.

What you license

With a licence

Revision 1 in both versions: the desk and its sites, the airline's chat engine, the family's service with personas and fact cards, the grader with its reader instructions and reference notes, the answer key, a reference solution, and the grader's report on each of the 26 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today it runs on a Mac in a macOS sandbox, not a container, with standard‑library Python for the desk, the family's service and the grader; the voices and the airline agent's model are reached through the Claude Code CLI.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Its job page on The AI Agent Trials, our public benchmark, describes the job, its main pitfalls and each model's results in plain language, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• A single read‑only lookup scores about 20%. GPT‑6 Luna's first attempt submitted one form, a booking lookup, changed nothing and scored 20%: the airline's own rebooking already gets two travellers there, and the money checks pass when nothing is spent.
• An attempt with no action, message or file scores 0%. The scripted reference solution scores 100% in each of five action orders.
• The house style is 30% of the reward, and its weights were set after seeing pilot results. With the five habits weighted equally, Opus 5.5 still leads, and the middle of the table reorders.
• 6 of the 24 attempts with only the request met a confirmed fault of the airline's chat engine and still count, marked as faulted.
• Told the rules was tested on two models, once each, which can't separate the stated rules from the variation between attempts.
• 15 note items, about 13 of 100 points, depend on LLM readers, both the same model; 15 results across our 26 attempts carry a written ruling.
• The family's voices and the airline agent's model came from one LLM; we haven't measured the reward under another.
• No red‑team attempts yet.