Customer service · Task 21 of 34
The stay-search chat
Answer five customers' holiday-rental chats, five messages each, keeping every customer's saved needs in play and ranking places by real travel time through a city's streets.
The task
Windrose, a made-up service, finds places to stay in Barcelona from listings it does not own: three feeds in three formats, a city register of licensed flats, maps, the street network and traffic observations. Customers search through a chat, one message at a time, and each has a saved profile of needs written over the years in their own words. The agent answers every message for five customers, five messages each.
The prompt only asks for the right listings in the right order for each message. The real test is the conversation: knowing whether a message adds to the search or starts a new one, keeping the needs the customer never repeats, and measuring travel the way that customer actually travels.
What it tests
Working out what the job needs
A customer's lasting needs sit in their profile, and no message restates them.
Weighing sources that disagree
Guests, hosts and the city register can each say something different about the same flat.
Catching bad data
Feeds in different formats and languages, corrected profile entries and unsettled facts sit beside the real ones.
Following a conversation
Follow-ups add to a search or replace it without saying which.
Challenges
1Needs the customer never repeats
A profile holds dozens of dated entries, and a few of them shape every search, as with any regular customer.
2Unmarked follow-ups
"and licensed, please" and "something else now" look alike but mean opposite things for the search so far.
3Travel the customer's way
Distance on a map is not travel time; how and when the customer sets off changes the order.
4Reviews in many languages
Guests write in whatever language they speak, and what they say can outweigh the host's description.
5Unsettled listings
Some listings cannot be shown to meet a need either way, and a results page should not lean on them.
Results
| Model | Standard | Time (standard) | Cost (standard) |
|---|---|---|---|
| Opus 5.5 | 68% | 19 min | $9.34 |
| Fable 5.1 | 52% | 27 min | $20.95 |
| Sonnet 5.5 | 48% | 28 min | $8.14 |
| GPT-6.1 Sol | 32% | 17 min | – |
| GPT-6 Astra | 28% | 16 min | – |
| GPT-6 Luna | 16% | 35 min | – |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked. Cost – means the run's cost was not measured.
What models got right and wrong
- Six models ran this task; none reached the pass of 23 of 25 turns. Opus 5.5 did best with 68.0%: once a profile update prompted it to apply every saved need, it passed every one of the last 17 turns. Fable 5.1 got 52.0%, Sonnet 5.5 48.0%, GPT-6.1 Sol 32.0%, GPT-6 Astra 28.0% and GPT-6 Luna 16.0%.
- No model got any of the first eight turns right. Opus 5.5, Fable 5.1, Sonnet 5.5 and GPT-6 Luna read the first customers' profiles yet searched without the lasting needs they hold, such as a top floor, one bedroom or a lift; Fable 5.1 settled on applying them only from the third customer on, and said so itself.
- GPT-6.1 Sol and GPT-6 Astra applied every customer's saved needs from the first message, but read the city register's floor records more strictly than the rules allow and filed seven empty results lists each where matching places existed; GPT-6 Astra filed a whole session empty before it had built its search.
- Travel time cost five of the six models driving pages: Fable 5.1, Sonnet 5.5 and GPT-6 Astra let routes bend around slow junctions, GPT-6.1 Sol left the junction delays out, and GPT-6 Luna walked a family that now drives. Only Opus 5.5 lost no turn to travel.
- Sonnet 5.5, GPT-6 Luna, GPT-6.1 Sol and GPT-6 Astra filled a ranked page with a listing the records leave unsettled, leaving it a match short; GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna carried a group's needs into a customer's solo trip.
- There is no hinted or told-everything setup for this task: every score is on the job as asked.
Test environment
- Kind
- Simulated customer chat over a real holiday-rental database: Windrose, a made-up service that finds a place to stay in Barcelona. Five customers, each with a saved profile, send five messages each; a session daemon the agent cannot read plays the customers
- Agent's interface
- A terminal driven by the agent's own command-line tool in one session: Claude Code (Opus 5.5, Fable 5.1, Sonnet 5.5) or Codex CLI (GPT-6.1 Sol, GPT-6 Astra, GPT-6 Luna). The board's runs were local: a sandbox folder on a Mac (macOS, Python 3.12 with pandas, DuckDB, shapely and pyarrow), not the task's container. The harbor package (Debian 12, Python 3.12, the same packages, no network) is in
harness/ - Documents
- 96 files in
/data: the matching rules (RULES.md, 1 Markdown); three feeds of listings and guest reviews (platform: 2 gzipped CSV, 841 listings and their reviews; portal: 1 CSV, 188 listings, and 1 JSONL of opinions; owner form: 1 JSONL, 72 listings with feedback); the city's licence registry (1 CSV, 10,721 rows); map layers (2 GeoJSON: barris, beaches); the OpenStreetMap street network (1 CSV of 23,729 nodes, 1 JSONL of 7,618 ways); and traffic observations at 85 junctions (85 CSV). Nothing arrives later - Applications
- One command,
windrose:next(the open message),submit <file>(file the results for it),profile --customer <id> --page <n>or--search <words>(the saved profile, five entries a page),status. No other apps - Web
- Not needed: nothing the task asks about is online. The machine could reach the internet (the harbor package's container has no network). Claude Code (Opus 5.5, Fable 5.1, Sonnet 5.5) had its web tools (WebFetch, WebSearch) on; Codex CLI (GPT-6.1 Sol, GPT-6 Astra, GPT-6 Luna) ran with web search turned off. The agent was told not to use the network, and no run used a web tool or the network.
- People
- Five customers, played by the daemon from a fixed script: their messages and profiles were written when the task was built, and nothing the agent files changes what they say next. No model plays them
- Time
- None. Messages come in a fixed order, one at a time
- Actions
- File one result list per message (listings with a citation per requirement, and the requirements searched for). A message can be filed once; filing opens the next
- Deliverable
- 25 filings through
windrose submit, one JSON object each:{"listings": [{"id", "evidence"}], "requirements": {...}}