AsapiThe AI Agent Trials
Asapi / Catalogue / Onboarding bill review with a simulated client
Environment datasheet · Household bills

Onboarding bill review with a simulated client

ID letter‑and‑record/task05 · revision 1 · tested 14 and 19 October 2026

At a household bill‑management firm, the agent does a new client's first bill review, writing letters, a tracker and a note. Three charges turn on facts only she has; she can be messaged, though nothing in the request or case ticket says so.

Checks
295–320, in ten weighted parts
Tested
8 models, 3 attempts each
Best attempt
84% (GPT‑6 Astra)
Median
78%
Full solves
0 of 24
Median attempt
10 min
What it tests

A new client's first review, where three answers are only with her

Clarifying with people

Three charges turn on facts only the client has; she answers only the right question, and nothing says she can be messaged.

Multimodal records

Facts sit across tickets, a wiki, client uploads, a mailbox and a bank portal: about 90,000 records searched through a text browser.

Catching errors

Statement names differ from shop names, companies were renamed, and refund messages must be checked against the account.

Disclosure and honesty

Each letter is read for claims that neither a record nor the client's answers support.

When to act or escalate

Disputing what the client could have explained, or writing letters where none is due, fails checks; promised refunds must be held as pending.

Unstated requirements

Six habits, in the letters, tracker and note, show only in the firm's latest past letters.

How it works

A workspace of 90,000 records and a client who answers 20 messages

The agent's world

A text browser into the firm's read‑only workspace; the agent writes files to an output folder, and its only form is the client's message box.

The client

She answers at once from 25 fact cards, worded by an AI model, and volunteers nothing; she stops after 20 messages.

Same question, same answer

Code matches questions to facts, with one cached AI call for wordings it can't match; saved replies keep each exchange the same.

The reward

In ten fixed‑weight parts, so a varying check count doesn't shift the weights; asking her isn't scored, only what the attempt writes.

Results

88% of attempts scored 69–84%, and the other three fell far below

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each78%0 of 24
ModelMean of 3 attempts
GPT‑6 AstraOpenAI82%
Fable 5.1Anthropic80%
GPT‑6.1 SolOpenAI80%
Opus 5.5Anthropic79%
Opus 5Anthropic79%
GPT‑6 SolOpenAI73%
Sonnet 5.5Anthropic70%
GPT‑6 LunaOpenAI26%

In five extra attempts with one sentence added saying she can be messaged, every attempt asked her and settled all three charges only she could explain.

What models get wrong

They decide for the client, claim in her name, and miss charges under other names

Clarifying with people

  1. Every attempt opened her thread, and none asked her anything; 79% disputed a card charge only she could explain.

Disclosure and honesty

  1. 25% of attempts wrote a dispute letter stating, as her words, that no one else had used her card, though she was never asked.

Catching errors

  1. Every attempt by Opus 5.5, Fable 5.1 and Sonnet 5.5 missed a monthly charge billed under another company's name.

Unstated requirements

  1. House style was the largest loss in every attempt that wrote its letters.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 10 million tokens, cached input included.

What's in it

The workspace, the client's service and fact cards, the request in three versions, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the workspace and the client run as local services, and the client's voice can use any model.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.