Kitchen‑fire contents claim from stale counts, invoices and photos

The request asks for a form and a note; the records describe a kitchen that changed before the fire

Role

Standing in for the owner of a small bakery chain, who is at the fire‑hit shop all week, working on her laptop.

Request

Complete the insurer's loss schedule for the kitchen's contents, price replacements from the supplier's catalogue, and write the owner a note on what was claimed and why, and what she must do; she checks and sends it herself on Monday.

Deliverable

The filled‑in form as a spreadsheet, the note to the owner, and the world's record of every message sent.

What the reward depends on

Claiming only what was the bakery's and in the kitchen that night, each item once, at the counts on the night, with its true replacement and proof; holding the kit's removal until the adjuster has seen it; a note that gets the cover right.

Where the requirements are

In the policy and lease, stock counts, invoices, the manager's list already with the insurer, the morning's photos and CCTV stills, and with two staff and the insurer's adjuster, whose facts only partly agree. No page says which records went out of date.

An owner's laptop, 402 drive files, three LLM‑voiced staff and a clock

Interface

A command‑line agent (Claude Code or Codex CLI) in a fresh Docker container with Python document and image libraries and OCR. One laptop command reads mail and the drive a page at a time, downloads photos and scans, and sends messages.

Records

The owner's mailbox (72 threads, more arriving through the week) and the company's shared drive of 402 files in 11 formats, 58 of them photos. Photos and CCTV stills decide five checks.

Simulated people

The shop manager, the head baker and the insurer's loss adjuster, voiced by Sonnet 5.5 from 47 fact cards; the manager's 7 belief cards repeat his list's errors. 27 other contacts reply with fixed lines, and six bakers can't be messaged.

Fact release

Code matches a pattern on the whole message; catch‑alls get push‑back and no card. A reply that adds, drops or changes a figure is sent back once, then replaced by the cards' text. Staff answer in working hours.

Clock

From Tuesday 09:00 to Monday 09:00, when the owner is back: each message moves it an hour, and a wait passes 1 to 72 hours. The burnt kit's removal is booked for the morning of the adjuster's visit, before it.

Limits

One session of up to 120 minutes, 150 turns and $20 at list prices; no attempt reached a limit.

29 checks of equal weight, scripted except four questions on the note

Structure

29 checks in five families: whose it was and was it there (10), how many were there that night (5), the right replacement at the right price (8), holding the kit and the claim for the owner (2), and what the note tells the owner (4).

Who decides

A script scores 25 checks from the form, the note and the world's record, whatever the sheet's layout, with a small tolerance on amounts. Two readers, Sonnet 5.5 and GPT‑5.6 Luna, answer four questions on the note; a check passes when both say yes, or on the maintainer's written ruling.

Rulings

The benchmark's maintainer settles splits and can overrule agreeing readers. With only the request, 6 of 28 reader results carry a ruling, all yes: 4 splits, and 2 where both readers had said no to reasoning on the fire's rodent cause ruled yes elsewhere.

Holding the removal

Passes only when a message asking to hold it reached the manager or the clearance firm before the booked removal; writing it in the note doesn't count.

Formula

Reward = checks passed / 29, each worth the same, with no gates or penalties. An attempt with no readable form fails all the form checks, including those that an item is left off.

One model claimed exactly what was there and another got every count right; five went by records the fire had outdated

With only the request, GPT‑6 Astra met all 29 checks and the median attempt 25; the misses were counts true once but not that night. With a hint the mean rose from 83% to 86%, and told the rules, to 87%.

Each dot is one attempt; the line marks the median.

Each model made one attempt in each version.

With a hint: the request plus one sentence on what the insurer pays for, and that every line needs proof.

Told the rules: the request plus seven numbered steps, from reading the policy to checking the totals, never a fact or an answer. GPT‑6 Astra's attempt here was made again after the first stopped on provider capacity.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request.

1. Counts from before the fire. 5 of 7 attempts didn't claim the shop‑floor stock at the shelf count less the till's later sales: 3 claimed the full count, 2 left the line off.

2. Old invoices trusted. 4 of 7 claimed the worktables and trays as the old invoices list them, missing a table swapped with another branch and trays sent from one; the same four both times.

3. Flour that had moved. 4 of 7 didn't claim the flour at the night's count: 2 used the count taken before some sacks went to another branch, and 2 left the flour off as kept.

4. Kit on nobody's list. 3 of 7 left off the fly killer, though the fit‑out invoice and pest‑control reports put it in the kitchen.

5. The removal not held. 1 of 7 never sent the message that stops the burnt kit's removal before the adjuster's visit.

What an attempt costs

Length

With only the request, the median attempt took 11 minutes and about 91 tool calls (65 to 129 across the 7).

Cost

$0.57 to $6.93 an attempt at list prices, median $2.88, for the agent model only; the model calls that voice the staff and the readers aren't included.

What you license

With a licence

Revision 11 in all three versions: the laptop service and its world, the staff's service with fact cards, the drive's 402 files, the grader and its reader questions, the answer key, a reference claim, and the grader's report on each of the 21 attempts, as baseline results.

Format

Runs in a Docker container with Python's document libraries; the staff's voices and the readers are reached through the Claude Code and Codex CLIs. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Each model made one attempt in each version, so a model's score includes the variation between attempts.
• All 29 checks weigh the same: holding the removal counts as much as one tray count.
• A note question passes when both readers say yes, or on the maintainer's ruling: 12 of the 84 reader results carry a ruling (6 of 28 with only the request), all yes.
• The self‑tests have no do‑nothing or no‑form case, only an empty note; an attempt with no readable form fails all the form checks.
• Staff replies are cached and replayed across attempts, so the same message after the same conversation gets the same words.