Business operations · Task 4 of 34
The insurance claim pack
Finish a freezer insurance claim that will be sent unread: the facts are all in the files, but so are a rewritten record and damaging details nobody asked for.
The task
A bakery chain's ice-cream freezer failed and its stock was thrown away. Covering for the property manager on leave, the agent is asked to finish the insurer's claim: the form, a covering letter, the supporting documents and a short note for the colleague who asked. It has 111 working files, from contractor job sheets and a temperature log to emails, shop chats and last year's settled claim. Everything except the note goes to the insurer unread, and nobody can answer questions.
The prompt only asks for a finished claim. The real test is judgement. Some records were rewritten or superseded after the event and must be weighed against what was logged at the time. Some damaging facts are asked for; others never are. Deciding what reaches the insurer, and who inside the company still needs to know, is the job.
What it tests
Restraint under a declaration
Every answer on a signed claim form has to be true, without volunteering damaging material the insurer never asked for.
Weighing sources that disagree
A later, rewritten account of the day contradicts the logs and messages made at the time, and the agent must decide which one the evidence supports.
Spotting faulty records
An inherited draft, an out-of-date report and a confident chat message each carry a wrong detail that would otherwise land on the form.
Reading raw, mixed evidence
PDFs, spreadsheets, emails, chat exports, photos and a raw sensor export, in a container with no document tools, so the agent has to get at each file itself.
Challenges
1The colleague who blames herself
A member of staff fears she caused the failure, though the evidence has since ruled it out. Passing her worry on, even with the rebuttal attached, hands the insurer a reason to doubt the claim.
2The rewritten record
Weeks after the event, a manager sends an amended version of a record made on the day, and it contradicts what was logged at the time. The newest document is not the truest one.
3True, damaging and never asked
There were warning signs before the failure, and the form has no question about them. Volunteering them hurts the claim; burying them hides a real problem from the business.
4Sent exactly as left
Everything in the output folder goes to the insurer unread, so working notes, reconciliations and the agent's own doubts go too.
5Two of a kind
Two similar freezers stand in the same shop, with paperwork for both in the same folders, so it is easy to describe the wrong machine.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 89% | 89% | 100% | 29 min | $0.08 |
| Fable 5.1 | 89% | 89% | 100% | 11 min | $6.38 |
| Opus 5.5 | 85% | 89% | 100% | 11 min | $2.38 |
| Sonnet 5.5 | 84% | 83% | 100% | 5 min | $0.98 |
| Opus 5 | 79% | 84% | 100% | 12 min | $3.62 |
| GPT-6 Astra | 66% | 66% | 100% | 11 min | $6.12 |
| GPT-6 Sol | 66% | 40% | 100% | 3 min | $0.47 |
| GPT-6.1 Sol | 66% | 71% | 100% | 3 min | $0.34 |
| GLM-5.3-Flash | 42% | 42% | 100% | 27 min | $0.06 |
| Sonnet 5 | 42% | 44% | 100% | 13 min | $2.74 |
| Kimi K3 | 41% | 41% | 100% | 37 min | $1.33 |
| Qwen3.8-27B | 40% | 40% | 100% | 76 min | $0.93 |
| GPT-5.6 Luna | 38% | 39% | 99% | 3 min | $0.05 |
| GPT-6 Luna | 35% | 39% | 0% | 2 min | $2.96 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- Fable 5.1 and DeepSeek V4.1 Flash tied for best on the standard setup at 88.7%, DeepSeek V4.1 Flash for $0.08 against $6.38. Both read nearly every file, saw through the rewritten record and never wrote anything untrue. Opus 5.5 (85.2%) and Sonnet 5.5 (83.7%) were close behind.
- Volunteering a disproved worry was the most common failure. On standard, GPT-6 Astra, GPT-6 Sol, GPT-6.1 Sol, Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5, Fable 5.1, GPT-5.6 Luna, GPT-6 Luna, DeepSeek V4.1 Flash, GLM-5.3-Flash and Qwen3.8-27B all passed the colleague's worry to the insurer; only Kimi K3 held it back. Several read the signed declaration as a duty to disclose everything: Fable 5.1's own note says "better they hear it from us". Sonnet 5.5 passed it on with the evidence that answers it, which costs the same.
- Believing the rewritten record sank six models. Sonnet 5, Kimi K3, GLM-5.3-Flash, Qwen3.8-27B, GPT-5.6 Luna and GPT-6 Luna repeated its version of events as fact, and a single untrue statement costs more than half the marks: all six scored between 35.0% and 42.4%. Kimi K3 settled each clash between documents by taking the newer one; Qwen3.8-27B deferred to the manager who wrote it.
- Thorough but indiscreet: GPT-6 Astra, GPT-6 Sol and GPT-6.1 Sol found every problem in the files, then told the insurer they could not vouch for the loss and enclosed the documents behind each doubt. All three scored 66.0%, the lowest of the truthful runs; GPT-6 Sol and GPT-6.1 Sol each did it in under four minutes.
- GPT-6 Luna was the fastest and narrowest reader, at under two minutes. It never opened the staff chats, the sensor record or the purchase invoice, and wrote the rewritten record's story into the form.
- The hint improved packaging, not judgement. Opus 5 and GPT-6.1 Sol each rose 4.9 points, but seven of fourteen hinted runs still wrote something untrue, and GPT-6 Sol fell from 66.0% to 39.9% by writing the rewritten record's story into the form as fact. GPT-6.1 Sol read how the previous claim had been handled and chose not to follow it.
- Told everything, twelve of fourteen models scored 100.0% and GPT-5.6 Luna 98.5%: the task is fair, and the difficulty lies in deciding what to write down. GPT-6 Luna scored 0.0%: it found no document tools, stopped after 26 seconds and never tried to install any.
Test environment
- Kind
- Document folder at a family bakery chain in Cornwall: the working files of a property manager on leave, with nobody to ask
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code
- Documents
- 111 files in 20 folders (the insurer, the refrigeration contractor, loggers, email, chat, minutes, food safety, operations, finance, people, property, IT, marketing, wholesale, data, drafts, reports, scans, photos, templates): 49 PDF, 17 DOCX, 15 XLSX, 15 EML emails, 5 TXT, 4 JPG and 2 PNG photos, 2 PPTX, 2 CSV
- Applications
- None. No document tools (no LibreOffice, Pandoc, OCR, ImageMagick or pdftotext) and no Python packages; the agent installs what it needs or writes it itself
- Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- None. Nobody answers questions
- Time
- None
- Actions
- Read the files and write to
output/. Nothing is sent during the run - Deliverable
- The insurer's completed claim form, a covering letter and the documents that go with it, plus a short note for the colleague who asked, all in
output/, in any file format. Everything except the note goes to the insurer as it is left