Medical bills review across two health plans

A couple's plan‑year medical bills across two health plans, handled the firm's way

Role

The review desk of a household bill‑management firm on a Sunday evening, covering an absent account manager: a married couple's medical and pharmacy bills for the plan year.

Request

Work one ticket: track every open bill and claim, record when each person's deductible and out‑of‑pocket maximum were met, and write the letters and a note to the couple, in the formats given. Message them whenever needed, without checking with anyone.

Deliverable

Letters with To, Re and Amount lines and at most 350 words, a tracker with nine statuses, a plan‑year status file and a note of at most 1,800 words.

What the reward depends on

The plans' rules (both administrators' costs counted together, two plans coordinated, pharmacy and spending‑account rules); handling each item as the firm does, including what an employer may be told; problems nobody flagged; what only the couple knows; the house style.

Where the requirements are

The plans' rules are in their documents. The firm's practices and habits show only in five past review packs of other households and its latest letters. Some problems sit in single records nobody flagged, and two facts are only with the couple.

82,283 records and two simulated spouses in one message thread

Interface

The text browser and nine desk tools of the other bill reviews, on one Mac: Claude Code 2.1.292 for the Anthropic models, Codex CLI 0.162.1 for the GPT models. The workspace's only form that changes anything is the send box on a client's thread.

Scale

82,283 records for 300 households: 7,257 tickets and 689 wiki pages, 76 of them past letters and five past medical review packs, and for the couple 106 files in eleven folders, a shared mailbox of 2,449 messages and 347 card transactions.

The plans

The husband's employer plan covers both, its medical and pharmacy claims run by two administrators that each print their own running total. The wife also has her own employer plan, and there are a spending account, a card and the firm's own bill pay.

Finding things

No list of records. The header search covers tickets, wiki, documents and feeds, not mail, the card portal or messages; quoted text in emails stays folded until expanded.

Simulated couple

Two spouses in one thread; a reply carries each spouse's own signed lines. They answer from 23 fact cards, a document line and 7 fact‑free default lines. The three other clients the desk holds answer one fixed line.

Fact release

Code releases cards and decides which spouse speaks. Questions whose answer is the review's own, or a plan rule, get a fact‑free line; catch‑all and injection‑style questions release nothing. One cached classifier call handles a paraphrase no route matches.

Voice

Sonnet 5.5 words each spouse's released lines; a part that adds, drops or changes a fact is replaced by the cards' text. They answer 20 messages an attempt, and the same exchange gets the same saved reply.

Clock

A fixed Sunday evening. Only the time stamped on a sent message moves, capped at 9:59 pm.

Sandbox

Each harness in its sandbox, as in the other bill reviews, with the couple's service outside them all; probes checked every attempt's sandbox first.

351 checks weighted by kind of challenge, one charge per error

Structure

351 checks, each tagged with the kind of challenge it tests; the reward is weighted by kind, not count. A check that can't apply drops out, its weight staying within its kind: 287 to 350 counted in our attempts with only the request.

Weights

The plans' rules 22%, the firm's practices 22%, problems nobody flagged 22%, house style 16% (four groups of 4), the couple's own items 11%, what only they know 3%, file form 4%. No check outside house style weighs more than 1 point.

One error, one charge

Where an answer depends on one the attempt got wrong, it's graded against what follows from that error (10 cascades), so one missed root problem isn't charged twice.

Messages

Whether an attempt messages the couple isn't scored; their answers count only through what it writes.

Who decides

Scripts decide 329. Two blind LLM readers, Opus 5.5 subagents, read 21 note items with three reference notes of known verdicts in every batch. The task's coordinator reads one check on unsupported claims on the letters.

Reader agreement

The readers matched every known verdict in every batch and split on 29 of 546 judgements (5.3%), each settled by the coordinator's written ruling, 28 of them to a fail.

Grader self‑test

The reference answer passes all 351 checks, 132 near‑miss answers built on a synthetic key each fail exactly the checks aimed at them, and an empty attempt scores 0. Fairness gates, all passing, show every deciding record and card is reachable.

Formula

Reward = the weighted share of applying checks passed, each kind scaled to its weight, from 0 to 1.

The four leaders lost under a point on the plans' rules; the firm's practices and style cost most

With only the request, 21 of 24 attempts scored 65–88%; GPT‑6 Luna's three ended within a minute at 0–11%. None fully solved it. Told the rules, Opus 5.5 scored 98%, and GPT‑6.1 Sol 99%, one check short of a full solve.

Each dot is one attempt; the line marks the median.

Three attempts per model with only the request, which says the couple can be messaged; one each told the rules, by Opus 5.5 and GPT‑6.1 Sol.

Told the rules: the request plus a paragraph naming every rule, practice and house habit and where to look, never an answer.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Care details to an employer. All 21 attempts that wrote letters described the husband's care in the letter to his employer, which the firm's past reviews never do. Both attempts told the rules kept it out.

2. A letter the firm wouldn't send. All 21 wrote to a surgeon's office about a bill the plan's own reprocessing will change, which past reviews don't do. Both attempts told the rules didn't.

3. A form printed in one record. 16 of 24 missed a form the plan needed back, printed only on one claim's statement. GPT‑6 Astra found it in all 3 attempts by searching for it; all 3 of Opus 5.5's missed it.

4. Payment timing. 16 of the 21 attempts that wrote letters didn't time new payments the way every past review does, losing 2.7 to 3.7 points; Fable 5.1 did in all 3 attempts, Opus 5.5 in 2.

5. House style. It cost 7.3 to 14.8 points an attempt, of the 16 it carries, and was the largest loss in all but two of the 21 attempts that wrote letters; only GPT‑6 Astra, in two, dated a call as the firm does.

6. Stopped early. GPT‑6 Luna ended each of its 3 attempts within a minute, saying the work was unfinished; none wrote a letter, and the first saved nothing.

What an attempt costs

Length

With only the request, the median attempt took 16 minutes and about 206 tool calls (19 to 423 across the 24), and about 14 million tokens, cached input included.

Cost

Recorded for 12 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the couple's voices and readers aren't included.

What you license

With a licence

The workspace with its five data files and desk tools, the couple's service and fact cards, the request in both versions, the grader with its readers' instruction and cascades, the fairness gates, the answer key and its reference answer, and the grader's report on each of the 26 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran on one Mac in each harness's sandbox, not in a container.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language. The workspace, fact cards, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model; the told‑the‑rules version was tested once each on two models, which can't separate the stated rules from the spread between attempts.
• The couple's voices were Sonnet 5.5 in every attempt, and once a passing mention released a fact nobody asked for (it is also in the mailbox). We haven't measured the reward under a second voice model.
• The note readers are Claude models (Opus 5.5), reading Claude and GPT notes; the reference notes measure their drift, not a possible preference for Claude's writing.
• A wiki page's wording on when to send payments can be read as the due date. The author kept the due date as a miss, costing those attempts 2.7 to 3.7 points each.
• Two challenges, forms printed only on two claims' statements, were added after pilot attempts read only the claims export; together they carry about 3.8 points.
• A check that doesn't apply leaves its weight within its group, so one house‑style sentence weighed 4 points in 7 of our 24 attempts with only the request and 2 in the rest; the author kept this by design.
• The weights were fixed after a first pilot, and the environment was built to keep its top models at or under 80%. Pilot attempts are on the board, and on the final scores the pilot's go rule wouldn't have been met.
• Doing nothing scores 0 and the reference answer passes all checks (the grader's self‑test). No red‑team attempts yet.