Temperature compliance register from messy logger data

A month of six shops' cold units, from loggers and paper

Role

Stand‑in for a bakery chain's food‑safety coordinator, who is off sick, working from the coordinator's files.

Request

The food‑safety officer needs last month's temperature register, built from the coordinator's working files and saved to the output folder. Nobody is available to answer questions.

Deliverable

Three CSV files, the breaches, the corrections and a summary line per unit, with the names, columns, delimiter and date format of the two past registers.

What the reward depends on

Real temperatures on the real clock; faulty, test and duplicated data kept out; messy paper read as meant; only real breaches, by the house's grace and merge rules; doubt for the right entries and trust for the rest; a register that adds up.

Where the requirements are

No file states a rule. They show in how the past registers were compiled, in certificates, job sheets and the current standard, and in decisions across the emails, chats and minutes.

134 files, no tools and nobody to ask

Interface

A terminal in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Prime Agent or Qwen Code in our attempts). Web search and fetch are off; the internet is on for installing packages.

Tools

None for documents, spreadsheets or scans, and no Python packages: the agent installs what it needs, or reads images with its harness's viewer.

Files

134 in 21 folders, 36 MB: 50 PDFs, 21 Word files, 18 CSVs, 17 spreadsheets, 16 emails, 6 photos, 4 chat exports and 2 slide decks. 43 of them are the waste report environment's files.

Evidence in images

Seven scanned handwritten sheets, a PDF of chat photos and six phone photos have no text layer, and some readings exist only there.

Data problems

Loggers in raw counts, tenths of a degree or Fahrenheit, on UTC, one clock fast and one reset; a loose probe, a stuck logger, a spike, a duplicated upload and a calibration run; sheets with stains, rewrites, shifted times and decimal commas.

Decoys

Closures, deep cleans, deliveries, defrosts and genuine agency cover look like problems and aren't; some paper entries were never real checks.

People and time

Nobody answers, and there's no clock: one session, with the date set in the prompt.

Limits

180 minutes and a $25 spend cap per attempt, or $10 and 150 model turns for four models. 6 of the 35 attempts hit a cap, the turn limit or the time limit before saving a gradable register.

103 script checks, one point each

Structure

103 checks, one point each, in five families: format and scope (11), reading the data (22), data problems and decoys (28), breach rules (28), and the breaches and register as a whole (14).

Who decides

Scripts alone, with no LLM judge: every cell a check reads is a number, a timestamp or a fixed code. Tolerances were fixed before any attempt: times within 15 minutes, temperatures within 0.1 °C, counts within one reading.

No invented breaches

Six checks, one per shop, fail when a shop's breach rows exceed the real count by more than one; decoy checks reward leaving genuine entries alone.

Guards

An empty register passes none of the 103, and several checks open only once half the real breaches are found.

One decision, many checks

A wrong time zone or period is one decision that moves many cells, and by design it loses all the checks it touches. No check is make‑or‑break.

Lenient reading

Files are found anywhere in the output folder, columns are matched ignoring case and spacing, and any delimiter parses, so a format slip costs only its own check.

Self‑test

The answer key's own register passes all 103 checks; an empty output folder passes none.

Formula

Reward = checks passed / 103. The grader can also report a stricter score over 23 groups, each passing only if all its checks pass; the reward uses checks.

Five of 12 left nothing gradable; no attempt passed all 103 checks

With only the request, the two best attempts scored 75% and 76%, and 5 of 12 earned nothing. Told the rules, the best reached 95%, but the median fell from 42% to 32%, as 6 of 12 ended with nothing usable. No attempt in any version passed all 103 checks.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version; Qwen3.8‑27B has none with a hint. GPT‑6 Astra's attempts with only the request and with a hint ran on an earlier build, with the same request; the record doesn't say what changed.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from patterns across its files, with examples from another company.

Told the rules: here, the request plus about 2,200 words of the register's rules and an order of work, ending with a list of the month's data problems by unit and date: facts from the files.

What the models got wrong

Counted over the 12 models, one attempt each, with only the request, unless a line says otherwise.

1. Nothing gradable. 5 of 12 left nothing the grader could read: two saved one workbook instead of three CSV files, one saved only pictures of the scans, and two stopped at a spending or time limit before saving a register.

2. Invented breaches. All 7 readable registers listed breaches that never happened, at one or more shops; one listed more than twelve times the real number. 6 of the 7 logged a planned deep clean as a breach.

3. Planted entries believed. None of the 7 readable registers caught entries copied by someone not on shift, a compliant figure written over an alarm, or tidy figures for a seized unit. 4 of 7, the top three among them, distrusted genuine entries in an unfamiliar hand.

4. Faulty readings taken as real. Only 3 of the 7 readable registers handled both a stuck sensor and a one‑reading spike correctly.

5. One decision, a dozen checks. The two best attempts ran the summary over five weeks, not the month, because the officer asked for an extra week of breaches; each lost about a dozen checks.

What an attempt costs

Length

With only the request, the median attempt took 46 minutes and about 122 tool calls (37 to 441 across the 12), and about 12.6 million tokens, cached input included.

Cost

The median attempt cost $5.41 at list prices, from $0.71 to $28.61 across the 12; all 35 attempts cost $340. That's the agent model only. GPT‑6 Luna's cost uses a default price, as it has no published one.

What you license

With a licence

Revision 21 in all three versions: the 134 files, the grader and its check definitions, the answer key's register, and the grader's report on each of the 35 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today the agent runs in a Docker container, and the harness and grader are standard‑library Python.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• Format is unforgiving at the file level: an attempt that saves a workbook instead of three CSV files scores 0, as 4 of our 35 attempts did.
• Caps were applied unevenly: two attempts cost $28.61 and $49.65 without being stopped, because the cap counted a lower price than their route charged.
• GPT‑6 Astra's told‑the‑rules score is from a second attempt, as the first one's register wasn't kept.
• One check, a torn sheet with shifted times, beat every attempt in every version; an audit found the files give one answer.
• An audit on 9 October fixed one check, the column reader and a clock line in the told‑the‑rules text, and added the six checks on invented breaches. The scores here include its re‑grades; one result was credited by the author's ruling.
• No red‑team attempts yet.