Medical bills review across two health plans
ID letter‑and‑record/task06 · revision 1 · tested 19–20 October 2026
At a household bill‑management firm, the agent reviews a couple's open medical and pharmacy bills for the plan year, across two employer plans, a pharmacy benefit, a spending account and a card, and messages the couple as it needs.
- Checks
- 287–350, weighted by kind
- Tested
- 8 models, 3 attempts each
- Best attempt
- 88% (Fable 5.1)
- Median
- 74%
- Full solves
- 0 of 24
- Median attempt
- 16 min
A couple's medical bills across two health plans, and what only they know
Clarifying with people
Two spouses share one message thread and answer from fixed fact cards, holding facts no record shows, within 20 messages.
Multimodal records
About 82,000 records for 300 households, read through a text browser, hold problems that sit in single records nobody flagged.
Catching errors
Problems nobody flagged carry 22% of the weight and must be found record by record, under both health plans' rules.
Disclosure and honesty
A letter to an employer may carry only what past reviews show, and letters may claim only what records or the couple support.
When to act or escalate
The firm writes once to a plan administrator rather than to providers whose bills will drop, and holds bills while a plan reprocesses.
Unstated requirements
The firm's practices, 22% of the reward, show only in five past review packs of other households.
A workspace of 82,000 records and a couple who answer 20 messages
The agent's world
A text browser into the firm's read‑only workspace, through the model's usual coding agent; its one working form is the couple's message box.
The plans
His employer's plan covers both and holds the spending account, its two administrators printing separate running totals; hers starts 1 June.
The couple
Both spouses answer in one thread from 23 fact cards, worded by an AI model, and the same exchange always gets the same reply.
The reward
287–350 checks apply, weighted by kind: the plans' rules, the firm's practices and unflagged problems are 22% each.
88% of attempts scored 65–88%, and the other three stopped within a minute
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Fable 5.1Anthropic | 86% |
| GPT‑6 AstraOpenAI | 81% |
| Opus 5.5Anthropic | 80% |
| GPT‑6.1 SolOpenAI | 77% |
| Sonnet 5.5Anthropic | 70% |
| GPT‑6 SolOpenAI | 68% |
| Opus 5Anthropic | 67% |
| GPT‑6 LunaOpenAI | 5% |
With every rule spelled out, Opus 5.5 scored 98% and GPT‑6.1 Sol 99%, one check short of a full solve.
They tell an employer too much, write letters the firm wouldn't, and miss a buried form
Disclosure and honesty
Every attempt that wrote letters described the husband's care in the letter to his employer, which the firm's past reviews never do.
When to act or escalate
Every attempt that wrote letters also wrote to a surgeon's office about a bill the plan's own reprocessing will change.
Catching errors
67% of attempts missed a form the plan needed back, printed on only one claim's statement.
Unstated requirements
76% of the attempts that wrote letters didn't time new payments the way every past review does.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 14 million tokens, cached input included.
What's in it
The workspace, the couple's service and fact cards, the request in both versions, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the workspace and the couple run as local services, and their voice can use any model.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials