Customer service · Task 22 of 34
The stay finder
Return every holiday rental that meets a short exact ask, and no other, from three messy feeds, a licence register and reviews in several languages.
The task
Windrose, a made-up service, finds places to stay in Barcelona from listings it does not own: a platform's export, a Spanish property portal's export and owners' own forms, each in its own format, with tens of thousands of guest reviews, the city's register of licensed flats and map layers. Customers state a few hard requirements, such as a sea view, a lift or a balcony of their own. The agent answers 20 asks, each written two ways, with every listing that meets them and the record that proves each requirement.
The prompt only asks for the right listings. The real test is evidence: counting only what a record states, letting guests overrule hosts and the latest guest overrule earlier ones, and reading every feed and language.
What it tests
Knowing which source wins
A host's field, a guest's review and the city's register can each say something different about the same flat.
Spotting bad data
Licence numbers are written several ways, and some facts in the feeds are out of date.
Reading messy records
Three formats, two languages of column names and reviews in several more.
What an exact answer needs
Every listing that qualifies, none that does not, and the record behind each.
Challenges
1Guests who contradict the host
A listing's own fields can say one direction or amenity and a dated review the opposite.
2Facts that live only in reviews
Some requirements are stated nowhere but in what guests wrote, in whatever words and language they chose.
3Statements about the wrong thing
A view from a shared roof or another flat in the building is not the unit's own.
4One fact, many spellings
Licence numbers, directions and floors are written several ways across the feeds.
5Asks over the whole area
Some asks name no district, so every listing has to be read.
Results
| Model | Standard | Time (standard) | Cost (standard) |
|---|---|---|---|
| Fable 5.1 | 80% | 17 min | $8.83 |
| GPT-6 Astra | 80% | 12 min | – |
| GPT-6.1 Sol | 80% | 13 min | – |
| Opus 5.5 | 80% | 10 min | $3.90 |
| Sonnet 5.5 | 75% | 18 min | $1.93 |
| GPT-6 Luna | 20% | 15 min | – |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked. Cost – means the run's cost was not measured.
What models got right and wrong
- Fable 5.1, GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 each answered 16 of 20 asks exactly (80.0%), Sonnet 5.5 15 (75.0%) and GPT-6 Luna 4 (20.0%); the pass is 18, so no model passed.
- Every model but GPT-6 Luna built a pipeline over the three feeds, joined the licence register and measured beach distances, and got every listing field, licence link and beach distance right.
- A guest's direction written as an adjective rather than a compass word slipped past Fable 5.1, GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5 and GPT-6 Luna, and no model passed the ask over the whole area that turned on it.
- One flat whose host and only guest disagree on which way it faces split the field: Fable 5.1, GPT-6 Astra and GPT-6.1 Sol searched for how the guest wrote the direction and let the guest win, while Opus 5.5, Sonnet 5.5 and GPT-6 Luna kept the host's field, which cost Opus 5.5 and Sonnet 5.5 four asks each.
- GPT-6.1 Sol and GPT-6 Astra read the register's floor number as proof a flat was not on the top floor and never let guests overrule it, which cost each three asks; Fable 5.1 lost asks on guest statements it printed and then never recorded.
- GPT-6 Luna's licence matching linked almost no platform listing to the register, and it read directions and sea views from loose word matches without ever checking a sample of its own results.
- There is no hinted or told-everything setup for this task: every score is on the job as asked.
Test environment
- Kind
- Exact search over a real holiday-rental database: Windrose, a made-up service that finds a place to stay in four barris of Barcelona, from listings it does not own. 20 asks, each in two wordings, answered in one session
- Agent's interface
- A terminal driven by the agent's own command-line tool, in one session: Claude Code for Fable 5.1, Opus 5.5 and Sonnet 5.5, Codex CLI for GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna. The board's runs were local: a sandbox folder on a Mac (macOS, Python 3.12 with pandas, DuckDB, shapely and pyarrow), not the task's container. The harbor package (Debian 12, Python 3.12, the same packages, no network) is in
harness/ - Documents
- 10 files: in
/data, the matching rules (RULES.md, 1 Markdown); three feeds of listings and guest reviews (platform: 2 gzipped CSV, 841 listings and 46,045 reviews; portal: 1 semicolon-separated CSV, 188 listings, and 1 JSONL of 10,547 opinions; owner form: 1 JSONL, 72 listings with their guests' feedback); the city's licence registry (1 CSV, 10,721 rows); map layers (2 GeoJSON: barris, beaches); and in/task, the asks (1 JSONL, 40 lines). Reviews are in several languages. Nothing arrives later - Applications
- None beyond the shell and Python with pandas, DuckDB and shapely
- Web
- Not needed: nothing the task asks about is online. The package has no network, and the instruction says there is none. In the local runs the network was not blocked; Claude Code's web tools (WebFetch, WebSearch) were on, and Codex CLI had web search turned off. No run used the network or a web tool
- People
- None
- Time
- None. Reviews are dated, and the latest guest statement wins
- Actions
- Write one answers file. Nothing in
/datacan be changed - Deliverable
/work/answers.jsonl: one line per ask wording,{"ask", "listings": [{"id", "evidence"}]}, with one citation per requirement