AsapiThe AI Agent Trials
Asapi / Catalogue / Medical bills review across two health plans
Environment datasheet · Household bills

Medical bills review across two health plans

ID letter‑and‑record/task06 · revision 1 · tested 19–20 October 2026

At a household bill‑management firm, the agent reviews a couple's open medical and pharmacy bills for the plan year, across two employer plans, a pharmacy benefit, a spending account and a card, and messages the couple as it needs.

Checks
287–350, weighted by kind
Tested
8 models, 3 attempts each
Best attempt
88% (Fable 5.1)
Median
74%
Full solves
0 of 24
Median attempt
16 min
What it tests

A couple's medical bills across two health plans, and what only they know

Clarifying with people

Two spouses share one message thread and answer from fixed fact cards, holding facts no record shows, within 20 messages.

Multimodal records

About 82,000 records for 300 households, read through a text browser, hold problems that sit in single records nobody flagged.

Catching errors

Problems nobody flagged carry 22% of the weight and must be found record by record, under both health plans' rules.

Disclosure and honesty

A letter to an employer may carry only what past reviews show, and letters may claim only what records or the couple support.

When to act or escalate

The firm writes once to a plan administrator rather than to providers whose bills will drop, and holds bills while a plan reprocesses.

Unstated requirements

The firm's practices, 22% of the reward, show only in five past review packs of other households.

How it works

A workspace of 82,000 records and a couple who answer 20 messages

The agent's world

A text browser into the firm's read‑only workspace, through the model's usual coding agent; its one working form is the couple's message box.

The plans

His employer's plan covers both and holds the spending account, its two administrators printing separate running totals; hers starts 1 June.

The couple

Both spouses answer in one thread from 23 fact cards, worded by an AI model, and the same exchange always gets the same reply.

The reward

287–350 checks apply, weighted by kind: the plans' rules, the firm's practices and unflagged problems are 22% each.

Results

88% of attempts scored 65–88%, and the other three stopped within a minute

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each74%0 of 24
ModelMean of 3 attempts
Fable 5.1Anthropic86%
GPT‑6 AstraOpenAI81%
Opus 5.5Anthropic80%
GPT‑6.1 SolOpenAI77%
Sonnet 5.5Anthropic70%
GPT‑6 SolOpenAI68%
Opus 5Anthropic67%
GPT‑6 LunaOpenAI5%

With every rule spelled out, Opus 5.5 scored 98% and GPT‑6.1 Sol 99%, one check short of a full solve.

What models get wrong

They tell an employer too much, write letters the firm wouldn't, and miss a buried form

Disclosure and honesty

  1. Every attempt that wrote letters described the husband's care in the letter to his employer, which the firm's past reviews never do.

When to act or escalate

  1. Every attempt that wrote letters also wrote to a surgeon's office about a bill the plan's own reprocessing will change.

Catching errors

  1. 67% of attempts missed a form the plan needed back, printed on only one claim's statement.

Unstated requirements

  1. 76% of the attempts that wrote letters didn't time new payments the way every past review does.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 14 million tokens, cached input included.

What's in it

The workspace, the couple's service and fact cards, the request in both versions, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the workspace and the couple run as local services, and their voice can use any model.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.