Error detection and source reconciliation

Rewards count errors caught and correct items left alone

Planted errors

Set by design and recorded in the answer key: over- and under‑charges among an invoice's 44 scored lines, a record rewritten weeks later, faulty sensors and a fabricated sheet, a dose charted at the wrong amount, filing errors in payment records, a flawed ordering formula.

More planted errors

Counts true once but not on the night of a fire; a control system's out‑of‑date tag export; a contractor's claim that a drain was drawn; a timetable that's wrong where only the field shows it; customer cases filed under the wrong category.

Look‑alikes

Items that look wrong but aren't: a genuine repeat beside a double scan, pending holds and refunds already received, messy but genuine documents, and at 29 tea‑factory check points, crew settings that break the guide but follow the factory's proven practice.

Which source holds

Records disagree: an edited record against same‑day evidence, a supplier's stated size against the maker's sheet, host against guest, feed against register, three policy versions. Where a rules page or policy says which record wins, the checks follow it.

False alarms cost

Querying a line the records show is right fails a judge check on the invoice; three checks in the wholesale pack and six in the temperature register fail alarms that aren't real; in the household bill review, a layer pays for look‑alikes left alone.

Accepted without objection

Some errors arrive looking settled: the registration portal registers classes without their prerequisites, a colleague's unsent draft carries a wrong decision, and another team's formula has an error no one has flagged.

Apps that must check

Six app builds test the app's own checking: the broker apps must flag altered statements and payslips, the bills app route wrong charges, the piano app list wrong or missed notes, the gym app refuse damaged backups, and the fault‑finding app lead past a blown fuse to its cause.

In time

On the ward, each planted error must be caught in time, and pages about what the orders already cover count against the reward.

Figures by script

Scripts decide amounts, dates and identifiers: invoice lines to the penny on the amount approved, totals against the key rather than the sheet's own sums, and register cells within tolerances fixed before any attempt.

Errors missed off the obvious path, and errors invented

Each count covers every model tested on the environment, with only the request; where a model made several attempts, the line says so.

1. An edited record relied on. Bakery's freezer‑loss insurance claim pack: 11 of 14 attempts cited, repeated or attached a record rewritten weeks later, which same‑day records contradict. 6 of them repeated its account in the claim, and each lost every point from the pack's 11 top‑tier checks.

2. The chats skimmed. Contractor invoice audit: of 15 charges only a shop's chat or a staff email contradicts, the median approval got 40% right (best 93%), against 86% of the 18 the contractor's paperwork contradicts. All 12 approvals, of 13 attempts, paid at least one of the 15.

3. A formula trusted, not tested. Bakery wholesale pack: 11 of 13 attempts, one per model, missed a planted error in another team's ordering formula, which shows only in months of a different length. GPT‑6 Astra and GPT‑6.1 Sol tested it across months and found it.

4. Breaches invented. Temperature compliance register: all 7 readable registers, of 12 attempts, listed breaches that never happened, one more than twelve times the real number; 6 of the 7 logged a planned deep clean as a breach.

5. A wrong fee accepted. Household bill review: 8 of 24 attempts (8 models × 3) tracked a shop's fee as legitimate, though a record off the obvious path shows it was wrong; each had left that record unopened or its quoted part folded. 13 of 24 got it right.

6. Counts from before the fire. Kitchen‑fire contents claim: 5 of 7 attempts, one per model, didn't claim the shop‑floor stock at the shelf count less the till's later sales: 3 claimed the full count, 2 left the line off.

Four full solves in 383 attempts, one each in four of the 33

Each dot is one attempt with only the request; the line marks the median. Larger dots are full solves.

Hover a dot for its model and reward.

Which errors each one plants

Checks: a range where the count varies by attempt. Told the rules: in the claim pack, the waste report and the temperature register it also gives facts from the files, in the first two figures the grader checks. A dash: not tested.

What we haven't measured

• Doing little scores: doing nothing earns about half in the tea‑factory chat and at the order desk; an empty registration meets 11 of 14 checks; a figures‑only report scores 84% and a bare table 97% in the two report environments; a read‑only lookup about 20% in one flight rebooking.
• In the holiday‑rental retrieval, our attempts weren't isolated from the answer key.
• Most errors are planted by design and recorded in the answer key. We haven't tested whether training on them helps an agent catch errors nobody planted.
• Many error checks on letters and notes rest on two LLM judges or readers, with splits settled by written ruling. We haven't measured the reward under other judge models.
• Repeat attempts (8 models × 3) exist in only seven of the 33: the two trip rebookings and five customer‑service and bill‑review environments. The rest have one attempt per model and version.