Offline fault‑finding tablet app for data‑centre cooling units
ID laycode/task11 · revision 1 · results as of 2 November 2026
A data centre's facilities lead wants a tablet app that leads a new night‑shift electrician, by touch alone, from a cooling unit that's down to its root cause, the repair and a filled‑in report. The agent builds it from the hall's records and a training bench.
- Checks
- 79, 40 points on faults, 60 on the app
- Tested
- 7 models, one attempt each
- Best
- 87% (Opus 5.5)
- Median
- 64%
- Full solves
- 0 of 7
- Median attempt
- 27 min, 49 tool calls
A tablet app that leads a new electrician from a dead unit to its root cause
Agentic coding
The offline app is walked through 20 faults it never saw, and a fix counts only if the unit then runs cleanly.
Multimodal records
The drawings are PDFs read only by looking, and each test point the app names must match that unit's drawing.
Catching errors
A blown fuse usually hides a shorted coil or winding, and some causes show only in the alarm history or maintenance log.
Unstated requirements
Only 3 of 79 checks cite the request; the rest include basics such as cutting the power before any work on a unit.
12 files, a training bench of 12 units, and 20 hidden faults
The agent's world
A fresh container with the hall's 12 files, among them drawings readable only by looking, worked through the model's usual coding agent.
The bench
A training bench simulates each of the 12 units from its wiring, with meters, displays, alarms and 8 practice faults; it refuses the graded ones.
Grading
An operator model plays the electrician, one fresh session per hidden fault, told only the unit and that it's down; the app runs offline.
The reward
Each of 20 faults earns two points, fixed and its root cause named; 39 app checks share 60, from safe isolation to the report.
Five of the seven scored 47–68%; the best reached 87% and the lowest 38%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 87% |
| GPT‑6 AstraOpenAI | 68% |
| GPT‑6.1 SolOpenAI | 66% |
| Sonnet 5.5Anthropic | 64% |
| Fable 5.1Anthropic | 64% |
| Sonnet 5Anthropic | 47% |
| GPT‑6 LunaOpenAI | 38% |
With an IT specialist's 23 requirements added to the request, the mean rose from 62% to 72%, and none fully solved it.
They miss a loose wire, stop at the blown fuse, and leave causes unnamed
Catching errors
Every attempt missed a loose motor lead at one unit's drive output, neither fixing nor naming it.
Every attempt left the shorts behind the chilled‑water units' blown fuses unfixed: apps kept fitting new fuses, gave no next step, or left the fuse open.
Every attempt left at least 5 of the 20 root causes unnamed, and one app's reports gave only symptoms or lists of possible causes.
Agentic coding
One app's report screen could never be reached, so its walks named no root cause at all.
Unstated requirements
One attempt had resistance measured on units that weren't cut off from every supply.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $6.90 at list prices for the agent model; the operator's and judges' calls are extra.
What's in it
The hall's 12 files, the training and grading benches, the 20 hidden faults, all three versions, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the agent's container, the benches and an offline sandbox for the app run side by side.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials