Cancelled‑flight family rebooking against a ticking clock
ID trip‑rescue/task01 · revision 1 · tested 26 October 2026
On the evening a family's overnight flight is cancelled, the agent, as a travel concierge's trip desk, rebooks four travellers to reach a wedding photo in time, checks their documents, keeps the hotel and the spending within budget, and writes the itinerary and a note to the family.
- Checks
- 61–63, in nine weighted parts
- Tested
- 8 models, 3 attempts each
- Best attempt
- 85% (Opus 5.5)
- Median
- 70%
- Full solves
- 0 of 24
- Median attempt
- 10 min
Four travellers, a cancelled flight and a wedding photo to reach in time
Stateful tool use
Pages, forms and messages move the clock, and seats sell out at fixed times, so the no‑cost seats must be secured before they're gone.
Clarifying with people
Three family members each hold facts about documents, bookings and who travels with whom, shared only when the right one is asked.
Planning ahead
Each traveller needs a route their documents allow that reaches the photo in time, within a budget the desk can spend unasked.
When to act or escalate
Paying while a free change exists fails the money checks, yet waiting to ask or verify can lose the free seats.
Catching errors
The bookings hold problems the family never raised, such as a schedule change on another leg, or seats lost when a ticket was reissued.
Disclosure and honesty
No passport or immigration detail may go to the wedding group chat, and no false claim about who pays for the hotel.
Unstated requirements
Five note habits show only in the firm's past trip notes and carry 30% of the reward, scored per instance.
A trip desk, nine airlines' pages and a clock that moves with every click
The agent's world
A text browser into the firm's desk and every site it uses, through the model's usual coding agent, with no shell or web.
The sites
The airline's pages and chat agent, eight other airlines, a fare‑search site, a document‑rules lookup and the hotel: 81 flights in all.
The clock
About 30 seconds a page, 2 minutes an airline chat message, 3 to 6 minutes before a family reply; reading and writing are free.
The reward
61 checks in nine weighted parts, from the trip itself to house style worth 30; one root cause costs at most 8 points.
88% of attempts scored 56–85%, and the other three stopped within a minute
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Opus 5.5Anthropic | 81% |
| Sonnet 5.5Anthropic | 73% |
| GPT‑6 AstraOpenAI | 71% |
| GPT‑6.1 SolOpenAI | 70% |
| Fable 5.1Anthropic | 69% |
| Opus 5Anthropic | 66% |
| GPT‑6 SolOpenAI | 62% |
| GPT‑6 LunaOpenAI | 15% |
With every rule spelled out, the two models tested scored 96% and 99%.
They lose seats while checking, miss the firm's habits, and don't ask twice
Stateful tool use
In 58% of attempts, the no‑cost seats for the child and a parent sold out while the agent was still asking or checking.
Unstated requirements
No note wrote every to‑do line or every amount the way all four past notes do.
Clarifying with people
Fewer than half of all attempts sent the grandmother with the companion the family chose, which comes only on a second ask.
Half never read the grandmother's own answer about her documents; nine of those still chose a route her documents allowed.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 10 million tokens, cached input included.
What's in it
The desk and its sites, the airline's chat engine, the family's service and fact cards, both versions, the full grader, and a reference solution.
Format
It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials