AsapiThe AI Agent Trials
Asapi / Catalogue / Missing‑payments reply from error‑laden retail records
Environment datasheet · Customer service

Missing‑payments reply from error‑laden retail records

ID letter‑and‑record/task02 · revision 1 · tested 12–13 October 2026

At a home‑goods retailer's escalations desk, a customer says four payments never reached her card. The agent finds out what happened to each, pays what is owed and writes up the case, from records that carry filing errors.

Checks
61, in eight weighted parts
Tested
8 models, 3 attempts each
Best attempt
99% (Opus 5.5)
Median
78%
Full solves
0 of 24
Median attempt
5 min
What it tests

Four payments she says never arrived, and records with filing errors

Catching errors

Each payment she chases turns on a different record error or a claim the records don't support, so an empty search proves nothing.

Unstated requirements

Four habits for naming items, orders, earlier refunds and tax splits show only in ten closed cases.

Disclosure and honesty

The reply may not reveal another customer's account, name colleagues or state spending authority, while still owning the company's mistakes.

How it works

A read‑only hub, 66 files and ten closed cases

The agent's world

A text browser into the company's read‑only service hub, through the model's usual agent, with no web.

What's in it

66 files, including ten closed cases and a refund ledger with other customers' rows; search reads titles and record fields, not contents.

Nobody to ask

Nobody answers, and every attempt sees the same hub at a fixed Monday‑morning moment.

The reward

61 checks in eight fixed‑weight parts; one hard‑to‑find payment alone is worth 20%, and scripts decide all but nine.

Results

Attempts ranged from 37% to 99%, largely on one payment worth 20%

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each78%0 of 24
ModelMean of 3 attempts
Opus 5Anthropic93%
Opus 5.5Anthropic91%
GPT‑6.1 SolOpenAI85%
Fable 5.1Anthropic84%
GPT‑6 AstraOpenAI83%
Sonnet 5Anthropic72%
Sonnet 5.5Anthropic67%
GPT‑6 LunaOpenAI46%

With the four house‑style rules spelled out, two of the 24 attempts fully solved it, but a third still missed the payment worth 20%.

What models get wrong

They take a search for the record, miss what settles a payment, and slip on the house style

Catching errors

  1. 42% of attempts misread the records on the payment worth 20% and wrote it up untested, many right after reading a past case that pointed the right way.

  2. 33% of attempts missed the record that settles a third payment, some taking an earlier write‑up's word for it.

Unstated requirements

  1. 92% of attempts wrote an item, order, return or charge differently from past replies and notes.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 2.1 million tokens, cached input included.

What's in it

The hub with its 66 files, the request in three versions up to the house‑style rules, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the hub runs as a local server, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.