Multi‑turn holiday‑rental search chat with saved profiles
ID tb‑tasks/windrose‑sessions · revision 2 · results as of 17 November 2026
At a holiday‑rental search service, the agent answers five customers' chats of five messages each. For every message it files the listings that customer should see, in order, carrying needs from a long saved profile, from three listing feeds, a licence register, maps and street traffic.
- Checks
- 25, 1 per message, all decided by script
- Tested
- 6 models, one attempt each
- Best
- 88% (Fable 5.2)
- Median
- 34%
- Full solves
- 0 of 6
- Median attempt
- 18 min, 94 tool calls
Five customers' chats, each search carrying needs from a long saved profile
Multimodal records
Answers must join each customer's saved profile with three listing feeds in different formats and languages, a register and map layers.
Unstated requirements
Standing needs sit on saved profiles, never restated, and each search must carry them unless the customer sets one aside.
1,101 listings in three feeds, street traffic and five scripted customers
The agent's world
A shell with Python and data libraries and one chat command to read the open message, page a profile and file results.
The customers
Five played by a fixed script: the next message appears only once the current one is filed, and nothing filed changes it.
Profiles
45 to 49 dated entries per customer in their own words, some later corrected, read five at a time or by word search.
The reward
One check per message: unranked lists must hold every proven match and no proven non‑match, ranked pages exactly the best five.
Attempts scored 8–88%, and the best missed 3 of the 25 messages
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.2Anthropic | 88% |
| Opus 5.5Anthropic | 60% |
| GPT‑6 AstraOpenAI | 40% |
| GPT‑6.1 SolOpenAI | 28% |
| Sonnet 5.5Anthropic | 28% |
| GPT‑6 LunaOpenAI | 8% |
They read profiles without applying them, trust the register's floor, and get travel times wrong
Unstated requirements
In half the attempts, the agent read the profiles, then searched without the standing needs they hold, such as a top floor or a lift.
When a customer travelled alone, 67% still applied the group's usual needs to that trip, and failed that message.
Multimodal records
Two took the register's numbered floor as proof a flat isn't on the top floor, and filed 5 and 7 empty lists where fitting listings existed.
Two lost ranked answers that turn on drive times: one never read the traffic records, the other applied one hour's delays to the whole route.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 13 million tokens, cached input included.
What's in it
The 96 files, rules page included, the chat service with the customers' script and profiles, the grader and answer key, and a scripted answer.
Format
Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container with no network.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials