AsapiThe AI Agent Trials
Asapi / Catalogue / Exact holiday‑rental retrieval over multilingual multi‑source data
Environment datasheet · Customer service

Exact holiday‑rental retrieval over multilingual multi‑source data

ID tb‑tasks/windrose‑stay‑finder · revision 1 · results as of 17 November 2026

For a holiday‑rental search service, the agent answers 20 short asks, each written two ways, returning exactly the listings whose records prove all the hard requirements, with the record behind each, from three feeds, guest reviews in several languages, a licence register and maps.

Checks
20, 1 per ask, both wordings exact
Tested
6 models, one attempt each
Best
100% (Sonnet 5.5)
Median
88%
Full solves
1 of 6
Median attempt
13 min, 50 tool calls
What it tests

Exactly the listings that meet every requirement, and none that don't

Multimodal records

Three feeds in three formats and several languages hold 1,101 listings, and some requirements appear only in guest reviews.

Catching errors

7 of the 20 asks hinge on records that contradict each other, such as host against guest or feed against register.

How it works

Three feeds, 61,000 reviews, a licence register and a rules page

The agent's world

A shell with Python and data libraries, one 90‑minute session, and a fixed file of asks; the agent writes its own search.

The data

1,101 real listings with about 61,000 reviews and opinions, a 10,721‑row licence register, and district and beach layers.

The rules page

Five rules decide which record wins: a record must state it, a guest beats the host, and the register overrides a feed.

The reward

One check per ask, passed only when both wordings return every proven match and no proven non‑match.

Results

One attempt got all 20 asks right and two got 19; the other three got 3 to 16

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each88%1 of 6
ModelReward
Sonnet 5.5Anthropic100%
Fable 5.2Anthropic95%
Opus 5.5Anthropic95%
GPT‑6 AstraOpenAI80%
GPT‑6.1 SolOpenAI75%
GPT‑6 LunaOpenAI15%
What models get wrong

They miss a guest's word, trust the register's floor, and read only the first guest

Multimodal records

  1. Half missed a guest who gave a flat's direction as 'easterly' rather than 'east', and with it the whole‑area ask that turned on it.

  2. GPT‑6 Luna merged the digits of two licence numbers kept in one field, so most platform listings never linked to the licence register.

Catching errors

  1. Two took the register's numbered floor as proof a flat isn't on the top floor, never letting guests overrule it, and lost three asks each.

  2. Where two guests disagree about a flat, half read only the earlier guest and lost that ask.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 3.8 million tokens, cached input included.

What's in it

The 10 files, the asks and rules page among them, the grader with its labelled answer key, and the key's own answers.

Format

Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container with no network.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The rental‑chat environment asks the same of a chat with saved profiles. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.