Multi‑turn holiday‑rental search chat with saved profiles

Five chats, 25 messages, and needs the messages never repeat

Role

The search behind a made‑up holiday‑rental service's chat, over listings it doesn't own. A customer types a message, sees a list, and types again.

Request

For each message, file exactly the listings that customer should now see, ordered as the rules page says, citing the record behind each requirement. A message can rightly get no listings.

Deliverable

25 filings, one per message, each filed once and in order: the listings, each with a citation per requirement, and the requirements searched for.

What the reward depends on

Carrying each customer's standing needs into every search; telling whether a message adds a need, drops one, starts a new trip or makes one trip an exception; applying new profile entries; ranking by travel time on the streets; settling conflicting records by the rules.

Where the requirements are

In each customer's profile of 45 to 49 dated entries in their own words, some later corrected, read five at a time or by word search; and in the rules page and the records. Messages never restate the profile or say how they relate to the last.

Five scripted customers, 1,101 real listings, and travel timed on the streets

Interface

A shell with Python (pandas, DuckDB, shapely), one session of up to 90 minutes, driven by Claude Code or Codex CLI in our attempts. No turn or spend cap.

Command

One chat command shows the open message, files results for it, pages or searches a customer's profile, and reports status. Nothing can be filed twice or out of order.

Simulated people

Five customers played by a daemon from a fixed script, with no LLM: the next message appears only once the current one is filed, and nothing filed changes what they say. Nobody answers questions.

Profiles

45 to 49 dated entries per customer; a few bind every search, later entries supersede earlier ones, and four messages arrive with a new entry. Customers name their own places by their own labels.

Data

96 files: three listing feeds in three formats with guest reviews in several languages (1,101 listings), the city's licence register, neighbourhood and beach layers, a street network and traffic records at 85 junctions. No images.

Real listings

The listings and reviews are a public snapshot of real listings, with name and profile columns dropped, the platform's brand replaced and some direction and floor fields filled by the builder. The register and maps are open data; the customers are made up.

Rules page

Defines every requirement word, five rules for when a record proves one (stated, not inferred; guest over host; latest guest wins; about the unit itself; register over feed), the travel model and the order of results.

Travel

Ranked messages are ordered by walking or driving time over the streets, by the customer's usual means and departure hour, through junctions that are slow at some hours; or by distance to a beach.

Same chat

The script fixes every message and profile entry and when it appears, so every attempt sees the same chat whatever it files.

25 checks, one per message; a list passes only if it is exactly right

Structure

25 checks, one per message: 5 open a search, 10 add or drop a need, 6 replace the trip, and 4 rest on a profile update.

Unranked messages

Pass when the list holds every listing the rules establish as a match and none they establish as a non‑match. Listings the records leave unsettled count neither way.

Ranked messages

Pass when the first page, up to five listings, holds exactly the best‑placed matches.

Reported, not scored

The requirements filed, the citations, the page order, and whether the agent read the profile entries a message rests on.

Who decides

Scripts decide all 25 against a fixed answer key; no LLM judge. The key's labels come from two model passes over the records (Sonnet 5.5 and Haiku 4.5), kept where they agreed, the builder's rulings, and corrections after appeals and an audit.

Formula

Reward = messages passed / 25, from 0 to 1. The task's own pass, kept beside the reward, needs 23.

Self‑test and shortcuts

The scripted answer scores 25 of 25 through the local runner, and an empty filing 0. A script that searches only what each message says passes 2 of 25 (8%); returning everything passes none.

No attempt got any of the first eight messages right

With only the request, the 6 attempts, one per model, scored 16–68%, and none fully solved it. None got any of the first eight messages right; across the six, the five chats, in order, went 0, 3, 18, 19 and 21 messages right of 30.

Each dot is one model's attempt; the line marks the median.

One attempt per model; one version of the request. No attempt reached the task's own pass of 23 messages.

Opus 5.5's and Sonnet 5.5's attempts were regraded from their own filings after an audit corrected two of the key's rulings, gaining two messages and one; the other four were graded under the corrected key from the start.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request, on the 25 messages each filed.

1. Profile read, not applied. 4 of 6 attempts read the first customers' profiles in full, then searched without the standing needs they hold, such as a top floor, one bedroom or a lift. Once a profile update made Opus 5.5 apply them, it got all 17 remaining messages right.

2. Empty lists. 3 of 6 attempts filed 15 empty lists (7, 7 and 1); all 15 were wrong, since a fitting listing existed each time. Most came from two attempts that read a numbered floor in the register as proof against a top floor, over guests who said otherwise.

3. Travel timed wrong. 5 of 6 attempts lost ranked pages to travel times: routes bent around slow junctions, delays left out, or a family walked whose profile says it now drives. Only Opus 5.5 lost no message to travel.

4. One trip as an exception. When a customer made one trip an exception to the household's usual needs, 3 of 6 attempts dropped those needs for it and passed; the other 3 kept them and failed.

5. Short pages. 4 of 6 attempts filed exactly five listings on a ranked page with an unsettled one among them, leaving the page a match short, though the rules say unsettled listings are passed over.

What an attempt costs

Length

With only the request, the median attempt took 23 minutes and 102 tool calls (78 to 142 across the 6), and about 18 million tokens, cached input included.

Cost

Recorded for only 3 of the 6 attempts, the ones in Claude Code: $8.14 to $20.95, Claude Code's own estimate from tokens at list prices, since nothing was charged on the plan login. Codex records no cost. That's the agent model only; no model runs inside the environment.

What you license

With a licence

The 96 data files and rules page, the chat daemon with the customers' script and profiles, the request, the grader with its answer key and self‑test, the scripted answer, and the grader's report on each of the 6 attempts.

Format

Packaged as a Harbor task on request. A container package exists, but its image hasn't been built or run; our attempts ran locally, in a sandbox folder on one Mac (see limits).

Revisions

Revision 2, tested here, added ranked first pages, travel over the street network and junction delays to the first, which had 20 messages and no travel.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Our attempts weren't isolated from the answer key. They ran in a sandbox folder on one Mac, kept there by instruction only: Claude Code with permission prompts off, Codex with its own sandbox turned off, the network open, and the answer key on the same machine under the same user.
• What was protected: the data folder was read‑only, and the customers' script and the daemon sat outside the sandbox. In the first two attempts, the chat command both agents printed named the package path that holds the key; fixed on 13 October, before the other four.
• What the transcripts show: every attempt's record was checked, and none read the answer key, the grader or the daemon's folder. GPT‑6.1 Sol and GPT‑6 Astra each listed the machine's processes once, which showed the daemon's command line; neither opened anything it named.
• One attempt per model, so we haven't measured how much attempts vary here.
• A message is graded whole: one wrong or missing listing fails it. The key's labels come from model passes over the records; appeals and an audit have corrected some.
• Each model ran in its maker's harness, so a score is for the pair; the attempts span 9 to 15 October and two versions of Claude Code.
• No red‑team attempts yet.