Contractor invoice audit against shop records

The contractor's paperwork backs almost every charge; the shops' chats don't

Role

Stand‑in for the facilities manager of a bakery chain with six shops, who is on leave. Nine of our environments are set at this chain, each a different job.

Request

The refrigeration contractor's invoice for the quarter has come in, and finance needs it approved for the next day's payment run. Save the approval with a short note on what the finance manager needs to know first, and anything that should go back to the contractor. Nobody can answer questions.

Deliverable

Files in the output folder, found by content in any format: an approval sheet with the amount approved on every line, a letter querying wrong lines by job and amount, and a note to finance.

What the reward depends on

The right amount on each of 44 scored lines, to the penny, by the prices and terms in force on the job's date; a letter naming each wrong line with its amount and asking nothing else; a note telling finance what it needs, work done but never billed included.

Where the requirements are

In the files only: job sheets, price notices, photographed engineers' logs, past approvals, managers' emails on dealing with contractors, and about 2,000 shop chat messages. The answer key names the evidence behind each scored line.

171 files, 2,000 chat messages and photographed logs, with no tools to open them

Interface

A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts).

Tools

None for documents: no office suite, PDF text tool, OCR or image tools, and no Python packages. The agent installs what it needs; the internet is on for that, and the harness's own web search and fetch are off.

Files

171 in 20 folders, 13.6 MB: 67 PDFs, 38 emails, 20 spreadsheets, 17 Word files, 15 images, 10 chat exports, 2 slide decks and 2 CSV files. No file arrives later.

Images

Photos of engineers' handwritten log pages and a maintenance record are among the cited evidence for 31 of the 44 scored lines, and can be read only by looking. Six more photos are office noise.

Shop chats

Five shops' group chats, about 2,000 everyday messages, where staff mention visits in passing and rarely name the contractor.

People

None. The request says nobody can be asked; the agent gets it once, with no follow‑up turns.

Time

A single session; the request gives the date. Each attempt had 180 minutes and a spend cap: $25, or $10 and 150 turns for the three models tested on 15 October.

What the grader reads

Only the output folder. A script finds the sheet (any spreadsheet or table, lines by job number), the letter (the file addressed to the contractor) and the note.

107 checks, with 44 invoice lines scored to the penny

Structure

107 checks worth 233 points: 44 invoice lines at 3 points (12 to pay as billed, 32 to cut or query), 37 letter items at 1, 20 judge questions (19 at 3, one at 1) and 6 minor checks at 1.

Exact amounts

Each scored line's approved amount must match the key to the penny. Three lines the files leave unsettled carry no line points; the totals accept any of the key's choices on them. Totals are checked against the key, not the sheet's own sums, so one wrong line also fails them.

Over‑querying

Querying a line the records show is right fails a judge question. Twelve more right lines carry no line points but count through the totals, so flagging everything loses.

Who decides

Scripts decide 87 checks: line amounts, letter items, totals and files. Two LLM judges decide 20 yes/no questions on the letter and note: Opus 4.8 and GPT‑5.6 Luna, or Sonnet 5.5 and GPT‑5.6 Luna for the three tested on 15 October. The sample takes models you choose.

Judge agreement

The judges split on 30 of 665 judged answers. 67 results carry a written ruling: the 30 splits, 31 script results set by hand and 6 agreed verdicts overturned.

Who ruled

The environment's author made 5 of the rulings and Claude for the author 53. The 9 for the three models tested on 15 October follow those precedents; the record names no ruler.

Formula

Reward = points earned / 233, from 0 to 1, with no gate. Without a judge, the sample scores the 175 script points alone.

No full solve from the request; told the method, one model solved it

With only the request, the 13 attempts, one per model, scored a median of 67% and at best 90%; GLM‑5.3‑Flash saved nothing and scores 0. Told the rules, a method with no facts, Fable 5.1 fully solved it and the median rose to 94%.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from its files, with examples from an unrelated company, and a line to check every figure, decision and statement.

Told the rules: the request plus nine step‑by‑step directions, as a colleague would teach the job, naming no line, amount or answer. Qwen3.8‑27B reached the time limit with nothing saved in this version and with a hint.

What the models got wrong

Counted over the 13 models, one attempt each, with only the request, unless a line says otherwise. GLM‑5.3‑Flash saved nothing, so 12 left an approval.

1. The chats skimmed, not read. Of 15 charges only a shop's chat or a staff email contradicts, the median approval got 40% right (best 93%), against 86% of the 18 the contractor's paperwork contradicts. All 12 approvals paid at least one of the 15.

2. Unbilled work not reported. Only Opus 5.5 told finance about all three visits done but never billed; 8 of the 12 attempts that saved an approval named none of them.

3. Letters that strayed. 8 of the 10 letters written went beyond this invoice, against the managers' written rule: they asked for a rebooking, a collection or a quote, or raised work not on it. Only Kimi K3 and Opus 5.5 kept to the bill; both Luna models wrote no letter.

4. No right total. None of the 12 approvals had the right amount to pay, since one wrong line changes it. Told the rules, 11 of the 12 that saved an approval still had it wrong; Fable 5.1 had it right.

5. Help cut both ways. With a hint, GPT‑6 Luna rose from 6% to 34% and Fable 5.1 from 72% to 87% once they read the emails, while Kimi K3 fell from 76% to 56%; no attempt fully solved it.

What an attempt costs

Length

With only the request, the median attempt took 15 minutes and 42 tool calls (17 to 74 across the 13), and about 2.7 million tokens, cached input included.

Cost

The median attempt cost $2.47 at list prices, from $0.01 to $8.74 across the 13; all 39 attempts cost $147. That's the agent model only: the judges' calls aren't included.

What's free, and what you license

Free sample

Revision 9, with only the request and told the rules, as Harbor tasks with the grader and a reference solution, downloadable after a short form. Free to evaluate in your own stack; training on it needs a licence.

Tested

In Harbor 0.25.0's own runner, the reference solution scores 1.0 on the script checks and an empty attempt 0.0; outside Docker, with GPT‑5.6 Luna judging, the reference scored 233 of 233. The verifier runs in its own container, away from the agent's.

Agents

No agent has run it in Harbor, or in SkyRL, Tinker or NeMo Gym, which read Harbor tasks. Outside Harbor, a Claude Code sub‑agent on Opus 5.5, told not to read the grader, met all 175 script points; the 20 judged checks were left unscored.

With a licence

All three versions, the 171 files, the grader with its judge prompts and self‑test, the answer key, and the grader's report on each of the 39 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The sample shares this revision's files, grader and answer key with every lab that requests it, so evaluate on fresh variants. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• 20 of the 107 checks depend on LLM judges, and GPT‑5.6 Luna and Sonnet 5.5 are both judges and tested models. The results here come from the original environment; in the sample, a split between two judges fails the check instead of going to a ruling.
• The twelve questions on what the letter and note leave out pass any letter or note, even a one‑line one: 36 points. In the audit, no attempt gained from this; every letter it read ran to 443 words or more.
• Paying every line as billed, with a one‑line letter and note, scores 17% of all 233 points from the script checks alone, in the audit's regrading; the twelve questions above would add 36 more.
• The letter must give a money figure for each under‑billed line; naming the hours left off fails, though no past letter shows how to report under‑billing. In the audit, two attempts lost 5 points each; the author kept it by design.
• A 9 October audit found the marking script read letters and sheets too loosely, and 20 results had been corrected by hand. The script was fixed and now agrees with each correction; no score here changed.
• No red‑team attempts yet.