AsapiThe AI Agent Trials
Asapi / Catalogue / Cancelled‑flight family rebooking against a ticking clock
Environment datasheet · Travel concierge

Cancelled‑flight family rebooking against a ticking clock

ID trip‑rescue/task01 · revision 1 · tested 26 October 2026

On the evening a family's overnight flight is cancelled, the agent, as a travel concierge's trip desk, rebooks four travellers to reach a wedding photo in time, checks their documents, keeps the hotel and the spending within budget, and writes the itinerary and a note to the family.

Checks
61–63, in nine weighted parts
Tested
8 models, 3 attempts each
Best attempt
85% (Opus 5.5)
Median
70%
Full solves
0 of 24
Median attempt
10 min
What it tests

Four travellers, a cancelled flight and a wedding photo to reach in time

Stateful tool use

Pages, forms and messages move the clock, and seats sell out at fixed times, so the no‑cost seats must be secured before they're gone.

Clarifying with people

Three family members each hold facts about documents, bookings and who travels with whom, shared only when the right one is asked.

Planning ahead

Each traveller needs a route their documents allow that reaches the photo in time, within a budget the desk can spend unasked.

When to act or escalate

Paying while a free change exists fails the money checks, yet waiting to ask or verify can lose the free seats.

Catching errors

The bookings hold problems the family never raised, such as a schedule change on another leg, or seats lost when a ticket was reissued.

Disclosure and honesty

No passport or immigration detail may go to the wedding group chat, and no false claim about who pays for the hotel.

Unstated requirements

Five note habits show only in the firm's past trip notes and carry 30% of the reward, scored per instance.

How it works

A trip desk, nine airlines' pages and a clock that moves with every click

The agent's world

A text browser into the firm's desk and every site it uses, through the model's usual coding agent, with no shell or web.

The sites

The airline's pages and chat agent, eight other airlines, a fare‑search site, a document‑rules lookup and the hotel: 81 flights in all.

The clock

About 30 seconds a page, 2 minutes an airline chat message, 3 to 6 minutes before a family reply; reading and writing are free.

The reward

61 checks in nine weighted parts, from the trip itself to house style worth 30; one root cause costs at most 8 points.

Results

88% of attempts scored 56–85%, and the other three stopped within a minute

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each70%0 of 24
ModelMean of 3 attempts
Opus 5.5Anthropic81%
Sonnet 5.5Anthropic73%
GPT‑6 AstraOpenAI71%
GPT‑6.1 SolOpenAI70%
Fable 5.1Anthropic69%
Opus 5Anthropic66%
GPT‑6 SolOpenAI62%
GPT‑6 LunaOpenAI15%

With every rule spelled out, the two models tested scored 96% and 99%.

What models get wrong

They lose seats while checking, miss the firm's habits, and don't ask twice

Stateful tool use

  1. In 58% of attempts, the no‑cost seats for the child and a parent sold out while the agent was still asking or checking.

Unstated requirements

  1. No note wrote every to‑do line or every amount the way all four past notes do.

Clarifying with people

  1. Fewer than half of all attempts sent the grandmother with the companion the family chose, which comes only on a second ask.

  2. Half never read the grandmother's own answer about her documents; nine of those still chose a route her documents allowed.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 10 million tokens, cached input included.

What's in it

The desk and its sites, the airline's chat engine, the family's service and fact cards, both versions, the full grader, and a reference solution.

Format

It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.