Temperature compliance register from messy logger data
ID house‑rules/task2‑1 · revision 21 · tested 21–22 September, 9 and 15 October 2026 · regraded 17 November 2026
Covering for a food‑safety coordinator who is off sick, the agent compiles a six‑shop bakery chain's monthly temperature register: every breach and every data fix for each fridge, freezer, display and hot cabinet, as three CSV files in the past registers' format.
- Checks
- 103, in 23 equal groups, each paying only if all its checks pass
- Tested
- 12 models, one attempt each
- Best
- 40% (Sonnet 5.5)
- Median
- 9%
- Full solves
- 0 of 12
- Median attempt
- 46 min, 122 tool calls
A month of temperatures from six shops, turned into a register
Multimodal records
Scanned handwritten sheets and phone photos hold readings found nowhere else, beside logger files in mixed units and clocks.
Catching errors
Genuine unusual entries must be trusted and extra breaches never invented, while faulty sensors, clock errors and a fabricated sheet are caught.
Unstated requirements
Past registers show the firm's breach rules, such as a short dip back in range not ending an excursion; they override a consultant's different rules.
134 files, loggers on different clocks and paper with no text layer
The agent's world
A terminal in a fresh container with 134 files and no document, spreadsheet or scan tools, through the model's usual coding agent.
Data problems
Loggers in raw counts or Fahrenheit, on UTC, one clock fast; a stuck logger, a spike, a duplicated upload and decimal commas.
Limits
Three hours and a spend cap per attempt, with nobody to ask.
The reward
103 checks in 23 equal groups, all by script; a group pays only if all its checks pass.
58% of attempts scored 6–40%; the rest left nothing the grader could read
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Sonnet 5.5Anthropic | 40% |
| Opus 5.5Anthropic | 40% |
| Fable 5.1Anthropic | 23% |
| Sonnet 5Anthropic | 15% |
| GPT‑6 AstraOpenAI | 13% |
| GPT‑6.1 SolOpenAI | 13% |
| DeepSeek V4.1 FlashDeepSeek | 6% |
| GLM‑5.3‑FlashZhipu | 0% |
| GPT‑5.6 LunaOpenAI | 0% |
| GPT‑6 LunaOpenAI | 0% |
| Opus 5Anthropic | 0% |
| Qwen3.8‑27BAlibaba | 0% |
Of the five that left nothing readable, two saved one workbook instead of three CSV files, one saved only pictures of the scans, and two stopped at a limit.
They invent breaches, believe planted entries, and miss the past registers' format
Catching errors
Every register the grader could read listed breaches that never happened, and six of those seven counted a planned deep clean, in whole or in part, as one.
None of the seven readable registers flagged a compliant figure written over an alarm.
Unstated requirements
The two best attempts ran the summary file over five weeks, not the month, which voids several groups at once.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $5.40 at list prices for the agent model.
What's in it
The 134 files, the request in all three versions, the grader and its check definitions, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; it runs in Docker today, and grading is a script alone.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials