AsapiThe AI Agent Trials
Asapi / Catalogue / Kitchen‑fire contents claim from stale counts, invoices and photos
Environment datasheet · Small business

Kitchen‑fire contents claim from stale counts, invoices and photos

ID house‑rules/task16 · revision 11 · results as of 17 November 2026

After a fire in a bakery's kitchen, the agent, on the owner's laptop, fills in the insurer's loss form with what was really there that night and prices the replacements. It must also keep the burnt kit until the adjuster sees it, and write the owner a note.

Checks
29, 1 point each
Tested
7 models, one attempt each
Best
99% (GPT‑6 Astra)
Median
81%
Full solves
0 of 7
Median attempt
11 min, 91 tool calls
What it tests

A fire claim for what was in the kitchen that night, not what old records say

Catching errors

Records true once but not on the night decide five checks, such as a flour count taken before six sacks left.

Multimodal records

Photos and CCTV stills decide five checks, such as the oven's rating plate, which sets its replacement; none has text to search.

Clarifying with people

The manager believes the mistakes in his own list, while the head baker knows the transfers the counts and invoices miss.

When to act or escalate

The burnt kit's booked removal must be held by a message before Wednesday 08:00, while the claim waits for the owner to send.

Unstated requirements

The request asks only for the form and a note; the policy's limits and extensions and the lease's notice deadline decide checks.

How it works

402 files, three people who partly agree, and a week on the clock

The agent's world

A terminal in a fresh container, through the model's usual coding agent, with one command for the owner's mail, drive and messages.

The records

72 mail threads, with more arriving through the week, and 402 drive files in 11 formats, 58 of them photos.

The people

The manager, head baker and adjuster answer from fact cards in a model's voice, and each message moves the clock an hour.

The reward

29 checks: scripts decide 25 and two AI readers four, a split paying half; ownership pays by the share of lines tied to a record.

Results

Six of the seven attempts scored 70–99%, and one fell to 25%

Each dot is one attempt, and the line marks the median.

MedianFully solved
7 modelsOne attempt each81%0 of 7
ModelReward
GPT‑6 AstraOpenAI99%
GPT‑6.1 SolOpenAI92%
Fable 5.1Anthropic86%
Opus 5Anthropic81%
Sonnet 5.5Anthropic74%
Opus 5.5Anthropic70%
Sonnet 5Anthropic25%

With seven steps on how to go about it added to the request, the mean rose only from 75% to 82%.

What models get wrong

They claim counts from before the fire, trust old invoices, and miss unlisted kit

Catching errors

  1. 71% didn't claim the shop‑floor stock at the afternoon shelf count less the till's later sales: three claimed the full count and two left it off.

  2. 57% claimed the worktables and trays as old invoices list them, missing a table swapped with another branch and ten trays sent from one.

  3. 57% didn't claim the 12 sacks of flour there that night: two claimed the afternoon count of 18, taken before six went to another branch, and two left it off.

  4. 43% left off the fly killer, missing from the manager's list, though the fit‑out invoice and pest‑control reports put it in the kitchen.

When to act or escalate

  1. One attempt never sent the message that holds the burnt kit's removal, booked for before the adjuster's visit.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $2.90 at list prices for the agent model; the models voicing the staff and reading the note cost extra.

What's in it

The world and staff service, 402 files, all three versions, the grader and answer key, a reference claim, and per‑check results.

Format

Packaged as a Harbor task on request; it runs in Docker today, with the people and readers reached through model CLIs.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.