AsapiThe AI Agent Trials
Asapi / Catalogue / Contractor invoice audit against shop records
Environment datasheet · Small business

Contractor invoice audit against shop records

ID house‑rules/task04 · revision 9 · tested 24 September and 15 October 2026 · regraded 17 November 2026

Covering for a facilities manager on leave, the agent approves a refrigeration contractor's quarterly invoice line by line for tomorrow's payment, writes the contractor a query letter and leaves finance a short note, working from 171 files with nobody to ask.

One of our three samples: Harbor tasks with the grader and a reference solution, free to download after a short form (19.7 MB).

Checks
107, worth 233 points
Tested
13 models, one attempt each
Best
61% (Sonnet 5.5)
Median
44%
Full solves
0 of 13
Median attempt
15 min, 42 tool calls
What it tests

A contractor's quarterly invoice, approved line by line for tomorrow's payment

Multimodal records

Photos of engineers' handwritten log pages are evidence for 31 of the invoice's 44 scored lines, among 171 files.

Catching errors

Planted over- and under‑charges sit among 44 invoice lines scored to the penny, and querying a correct line fails a check.

Disclosure and honesty

The contractor's letter must leave out staff names, chat quotes and requests beyond this invoice.

Unstated requirements

How approvals, contractor letters and the payment note are done shows only in past approvals and managers' emails.

How it works

171 files, photographed logs and 2,000 shop chat messages

The agent's world

A terminal in a fresh container with 171 files and no document tools, through the model's usual coding agent.

The evidence

Job sheets, price notices, photographed engineers' logs, past approvals, managers' emails and about 2,000 shop chat messages.

Limits

One session of 180 minutes with a spend cap, and nobody to ask; the request comes once, with no follow‑up.

The reward

107 checks worth 233 points; most pay only in proportion to how many key lines and totals the approval gets right.

Results

Nine attempts scored 27–61%, and four 16% or less

Each dot is one attempt, and the line marks the median.

MedianFully solved
13 modelsOne attempt each44%0 of 13
ModelReward
Sonnet 5.5Anthropic61%
Opus 5.5Anthropic58%
GPT‑6 AstraOpenAI53%
GPT‑6.1 SolOpenAI51%
Fable 5.1Anthropic48%
Kimi K3Moonshot44%
Opus 5Anthropic44%
DeepSeek V4.1 FlashDeepSeek31%
Qwen3.8‑27BAlibaba27%
Sonnet 5Anthropic16%
GPT‑5.6 LunaOpenAI1%
GPT‑6 LunaOpenAI1%
GLM‑5.3‑FlashZhipu0%

With a step‑by‑step method that gives no facts, the median rose to 78%, and the best attempt reached 95%.

What models get wrong

They skim the chats, get the total wrong, and stray beyond the invoice

Catching errors

  1. On 15 charges only a shop chat or staff email contradicts, the median approval got 40% right, against 86% on those the paperwork contradicts.

  2. No approval had the right total to pay, since one wrong line changes it.

Disclosure and honesty

  1. 80% of the letters went beyond this invoice, against the managers' written rule: five asked for a rebooking, a collection or a quote, and three raised other jobs.

Unstated requirements

  1. No note named all three visits done but never billed to both judges' satisfaction, and 67% of the approvals named none.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $2.50 at list prices for the agent model; the judges' calls are extra.

What's in it

All three versions, the 171 files, the full grader with its self‑test, and per‑check results for every attempt.

The sample

Harbor tasks in two of the three versions, with the grader and a reference solution, free for evaluation.

Format

In Harbor's own runner, the reference solution scores 1.0 on the script checks and an empty attempt 0.0.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

Its files and grader go out as a sample, so hold out fresh variants for clean evaluation; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.