Exact holiday‑rental retrieval over multilingual multi‑source data
ID tb‑tasks/windrose‑stay‑finder · revision 1 · results as of 17 November 2026
For a holiday‑rental search service, the agent answers 20 short asks, each written two ways, returning exactly the listings whose records prove all the hard requirements, with the record behind each, from three feeds, guest reviews in several languages, a licence register and maps.
- Checks
- 20, 1 per ask, both wordings exact
- Tested
- 6 models, one attempt each
- Best
- 100% (Sonnet 5.5)
- Median
- 88%
- Full solves
- 1 of 6
- Median attempt
- 13 min, 50 tool calls
Exactly the listings that meet every requirement, and none that don't
Multimodal records
Three feeds in three formats and several languages hold 1,101 listings, and some requirements appear only in guest reviews.
Catching errors
7 of the 20 asks hinge on records that contradict each other, such as host against guest or feed against register.
Three feeds, 61,000 reviews, a licence register and a rules page
The agent's world
A shell with Python and data libraries, one 90‑minute session, and a fixed file of asks; the agent writes its own search.
The data
1,101 real listings with about 61,000 reviews and opinions, a 10,721‑row licence register, and district and beach layers.
The rules page
Five rules decide which record wins: a record must state it, a guest beats the host, and the register overrides a feed.
The reward
One check per ask, passed only when both wordings return every proven match and no proven non‑match.
One attempt got all 20 asks right and two got 19; the other three got 3 to 16
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Sonnet 5.5Anthropic | 100% |
| Fable 5.2Anthropic | 95% |
| Opus 5.5Anthropic | 95% |
| GPT‑6 AstraOpenAI | 80% |
| GPT‑6.1 SolOpenAI | 75% |
| GPT‑6 LunaOpenAI | 15% |
They miss a guest's word, trust the register's floor, and read only the first guest
Multimodal records
Half missed a guest who gave a flat's direction as 'easterly' rather than 'east', and with it the whole‑area ask that turned on it.
GPT‑6 Luna merged the digits of two licence numbers kept in one field, so most platform listings never linked to the licence register.
Catching errors
Two took the register's numbered floor as proof a flat isn't on the top floor, never letting guests overrule it, and lost three asks each.
Where two guests disagree about a flat, half read only the earlier guest and lost that ask.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 3.8 million tokens, cached input included.
What's in it
The 10 files, the asks and rules page among them, the grader with its labelled answer key, and the key's own answers.
Format
Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container with no network.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The rental‑chat environment asks the same of a chat with saved profiles. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials