AsapiThe AI Agent Trials
Asapi / Catalogue / Multi‑turn holiday‑rental search chat with saved profiles
Environment datasheet · Customer service

Multi‑turn holiday‑rental search chat with saved profiles

ID tb‑tasks/windrose‑sessions · revision 2 · results as of 17 November 2026

At a holiday‑rental search service, the agent answers five customers' chats of five messages each. For every message it files the listings that customer should see, in order, carrying needs from a long saved profile, from three listing feeds, a licence register, maps and street traffic.

Checks
25, 1 per message, all decided by script
Tested
6 models, one attempt each
Best
88% (Fable 5.2)
Median
34%
Full solves
0 of 6
Median attempt
18 min, 94 tool calls
What it tests

Five customers' chats, each search carrying needs from a long saved profile

Multimodal records

Answers must join each customer's saved profile with three listing feeds in different formats and languages, a register and map layers.

Unstated requirements

Standing needs sit on saved profiles, never restated, and each search must carry them unless the customer sets one aside.

How it works

1,101 listings in three feeds, street traffic and five scripted customers

The agent's world

A shell with Python and data libraries and one chat command to read the open message, page a profile and file results.

The customers

Five played by a fixed script: the next message appears only once the current one is filed, and nothing filed changes it.

Profiles

45 to 49 dated entries per customer in their own words, some later corrected, read five at a time or by word search.

The reward

One check per message: unranked lists must hold every proven match and no proven non‑match, ranked pages exactly the best five.

Results

Attempts scored 8–88%, and the best missed 3 of the 25 messages

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each34%0 of 6
ModelReward
Fable 5.2Anthropic88%
Opus 5.5Anthropic60%
GPT‑6 AstraOpenAI40%
GPT‑6.1 SolOpenAI28%
Sonnet 5.5Anthropic28%
GPT‑6 LunaOpenAI8%
What models get wrong

They read profiles without applying them, trust the register's floor, and get travel times wrong

Unstated requirements

  1. In half the attempts, the agent read the profiles, then searched without the standing needs they hold, such as a top floor or a lift.

  2. When a customer travelled alone, 67% still applied the group's usual needs to that trip, and failed that message.

Multimodal records

  1. Two took the register's numbered floor as proof a flat isn't on the top floor, and filed 5 and 7 empty lists where fitting listings existed.

  2. Two lost ranked answers that turn on drive times: one never read the traffic records, the other applied one hour's delays to the whole route.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 13 million tokens, cached input included.

What's in it

The 96 files, rules page included, the chat service with the customers' script and profiles, the grader and answer key, and a scripted answer.

Format

Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container with no network.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.