Exact holiday‑rental retrieval over multilingual multi‑source data

20 exact asks, each written two ways: every match and nothing else

Role

The search behind a made‑up holiday‑rental service that promises every listing meeting a customer's hard requirements, and none that doesn't.

Request

Answer each ask with every listing whose records prove all its requirements and no other, reading the rules page literally and citing one record per requirement. An ask can rightly have no match.

Deliverable

One answers file: a line for each of the 40 ask wordings, each listing with one cited record per requirement.

What the reward depends on

Counting only what a record states; letting a dated guest review beat the host's field, and the latest guest win; linking listings to the register despite licence numbers written several ways; searching every feed and the whole area named; returning nothing when nothing fits.

Where the requirements are

Each ask names a place and two to four requirements from a closed vocabulary. The facts sit in feed fields, the register and guest text in several languages; the rules page says which record wins.

Three feeds, a register and 61,000 reviews, searched with the agent's own code

Interface

A shell with Python (pandas, DuckDB, shapely), one session of up to 90 minutes, driven by Claude Code or Codex CLI in our attempts. No service and no people: the asks are a fixed file, and the agent writes its own search.

Files

10: three feeds in three formats (a listing platform's export, a property portal's semicolon CSV with Spanish headers, owners' form entries) holding 1,101 listings and about 61,000 guest reviews and opinions, the register (10,721 rows), district and beach layers, the rules page and the asks.

Real listings

A public snapshot of real listings in four seafront districts of one city, with name and profile columns dropped and the platform's brand name replaced. Hosts' first names in review text were replaced on 13 October; all six attempts saw the original text.

Builder's changes

The portal's and owner form's direction fields were filled for 65% of their listings, four with a conflict planted against a dated guest review; floor fields were derived from the register and guests' statements.

Rules page

Defines every requirement word and five rules: a record must state it; a guest beats the host; the latest guest wins; it must be about the unit itself; the register overrides a feed where a licence links them.

Answer key

Every listing labelled for every requirement before any attempt as met, not met or unsettled: two model passes (Sonnet 5.5 and Haiku 4.5) kept where they agreed, plus the builder's rulings, then corrected in four rounds of appeals.

The asks

20, each written twice: 7 turn on facts found only in guest reviews, 4 on joining feeds to the register, 3 on conflicting records, 2 on structured fields, 2 span the whole area, and 2 have no match.

20 checks, one per ask; both wordings must come back exact

Structure

20 checks, one per ask. A wording passes when its list holds every listing the key establishes as a match and none it establishes as a non‑match; unsettled listings count neither way. An ask passes only when both wordings do.

Who decides

Scripts decide all 20 against the fixed key; no LLM judge, and no model is called while grading. Citations are checked and reported as a rate, not scored: bare listing ids with no citations score the same.

Formula

Reward = asks passed / 20, from 0 to 1. The task's own pass, kept beside the reward, needs 18.

Self‑test

The key's own answers score 20 of 20 (and scored 1 in Harbor on EC2, on the first key); an empty answers file scores 0.

Shortcuts

In the builder's calibration, a fields‑only search passed 6 of 20 asks (30%) and a keyword search 7 (35%). An empty answer passes only the two asks with no match; returning everything fails every ask with a non‑match.

Four attempts tied at 80%; none fully solved it

With only the request, 4 of the 6 attempts, one per model, got 16 of 20 asks exactly right (80%); Sonnet 5.5 got 15 and GPT‑6 Luna 4. The five best got every register link, capacity, beach distance and no‑match ask right; every attempt answered both wordings alike.

Each dot is one model's attempt; the line marks the median.

One attempt per model; one version of the request. No attempt reached the task's own pass of 18 asks.

Opus 5.5's attempt was regraded on the current key, gaining one ask. Sonnet 5.5's was rerun on 13 October on the current rules page; its first attempt had read an earlier page. The other four were graded on the current key directly.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request, on the 20 asks each answered.

1. The whole area. 0 of 6 attempts passed the whole‑area ask for a sea view facing one way: five never matched one guest's adjective for the direction, and the sixth found it and lost the ask on another listing. Whole‑area asks came back exact 3 times in 12.

2. Guest against host. On one listing, the host's field and the only guest to give a direction disagree: 3 of 6 attempts found the guest's compass word and let the guest win; Opus 5.5 never matched it, and Sonnet 5.5 printed it without acting, losing four asks each.

3. The register read as a denial. 3 of 6 attempts took the register's numbered floor as proof a flat isn't on the top floor, over guests who said they stayed there, though the rules say a numbered floor settles nothing. It cost GPT‑6.1 Sol and GPT‑6 Astra three asks each.

4. Loose matching. 1 of 6 attempts, GPT‑6 Luna's, linked almost no platform listing to the register, stripping every non‑digit from a cell holding two licence numbers, and counted a station's name and a remark about mould as sea views: 4 of 20 asks.

What an attempt costs

Length

With only the request, the median attempt took 14 minutes and 45 tool calls (37 to 51 across the 6), and about 3.4 million tokens, cached input included.

Cost

Recorded for only 3 of the 6 attempts, the ones in Claude Code: $1.93 to $8.83, Claude Code's own estimate from tokens at list prices, since nothing was charged on the plan login. Codex records no cost. That's the agent model only; no model runs at grading.

What you license

With a licence

The 10 files and the asks, the rules page, the grader with its answer key and labels, the key's own answers, and the grader's report on each of the 6 attempts.

Format

Packaged as a Harbor task on request. Its container package ran the key's answers in Harbor on EC2; our attempts ran locally, in a sandbox folder on one Mac (see limits).

Revisions

The key went through four rounds of appeals: where an agent's reading was upheld, it was corrected, and where the records don't decide, the listing was held unsettled.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Our attempts weren't isolated from the answer key. They ran in a sandbox copy of the files on one Mac, kept there by instruction only: Claude Code with permission prompts off, Codex with its own sandbox turned off, the network open, and the key on the same machine.
• What the transcripts show: every attempt's record was checked, and none read the key or opened anything outside its folder, apart from Sonnet 5.5 reading Claude Code's own record of a background command. The three Codex attempts listed the machine's processes, which named other folders; none opened them.
• What was protected: the sandbox held only the agent's files, nothing in it named the folder holding the key, and grading ran after the session ended.
• A model‑built key: its labels come from two model passes, and only the 10,439 of about 61,000 reviews a multilingual search proposed were labelled. In a 600‑review sample of the rest, 3 settled a fact.
• One listing's direction, set by one guest review against the host's field, decides 4 of the 20 asks. Read the other way, Opus 5.5 would have got all 20.
• One attempt per model, so we haven't measured how much attempts vary here; four attempts tied at 16 of 20.
• No red‑team attempts yet.