Offline fault‑finding tablet app for data‑centre cooling units

Touch‑only steps from a dead unit to its cause, its fix and the report

Role

Sole builder for a data centre's facilities team, after one repair followed the wrong drawing and a contractor bridged a safety switch.

Request

An offline web app for the shift's tablet: pick the unit, then tap through multiple‑choice steps that give each check, measurement and action, ending in a running unit and a completed root‑cause report.

Deliverable

The offline app and start notes for a clean machine with no network.

What the reward depends on

Each hidden fault fixed so the unit runs, with its root cause named; the right unit, drawing and healthy readings; cutting off every supply and waiting for stored charge to drain before work; a complete report; an app that works by touch alone.

Where the requirements are

In 12 files: three drawing sets as PDFs, one per unit model and readable only by looking, four parts lists, the controller manual, the fleet register, the maintenance log, the report form and the bench manual; touch‑only use and the no‑bypass rule come from the request.

12 files and a bench that simulates each unit from its wiring

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI). Internet is on for packages; web search and fetch are off.

The bench

A training bench that simulates each of the 12 units from its wiring: meter readings, displays, lights, alarms and simulated time, with 8 practice faults. It refuses the graded faults.

Grading

The app runs from its start notes with no network. An operator model, Opus 5.5 in the grader as shipped, plays the electrician, one fresh session per fault, told only the unit and that it's down, reading meter values from a separate bench loaded with the 20 hidden faults.

Limits

One build session of 120 minutes, 150 model responses and $20 at list prices. Each graded fault ends with a simulated restart and run.

79 checks: 40 on the faults, 39 on the app

Structure

Each of 20 faults has two checks, fixed so the unit runs and its root cause named, at a point each. 39 app checks share 60 points: the right unit, drawing and readings, working safely, and the report and the tablet.

Who decides

Scripts decide from the bench's log, the wiring and the simulator. Two LLM judges, Sonnet 5.5 and GPT‑5.6 Luna, read causes written in words and screens a word list flags as a possible safety bypass; the benchmark's maintainer ruled their one split and corrected 8 verdicts from saved evidence.

What counts

A report naming only the failed part, or several causes, fails; reading picks are checked against the bench; the graded faults exist only in the answer key and the grader's bench.

Formula

Reward = (fault points + 60/39 × app checks passed) / 100. An app that doesn't start from its notes with no network, or shows no screen, fails all 79 checks.

One app fixed 17 of 20 faults; none fully solved it in any version

With only the request, scores ran from 38% to 87%, the best ahead of the next at 68%. With a hint the mean rose from 62% to 66% but the median fell from 64% to 58%, as three of the seven scored lower; told the rules, the mean was 72%.

Each dot is one attempt; the line marks the median.

One attempt per model and version; 11 Claude attempts halted by subscription limits were made again from the start.

With a hint: the request plus 8 points from a senior electrician.

Told the rules: the request plus 23 product requirements from an IT specialist.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request.

1. A loose wire missed. 7 of 7 attempts neither fixed nor named a loose motor lead at one unit's drive output.

2. Stopping at the fuse. The shorts behind blown fuses on the chilled‑water units were fixed in 0 of 7 attempts: apps looped on fitting a new fuse, offered no reading or next step for the short, or named the short and left the fuse open.

3. Causes left open. No attempt named more than 15 of the 20 root causes. Two named none: one's reports gave only symptoms or lists of possible causes, and the other's report screen could never be reached.

4. Working on live units. 1 of 7 had resistance measured on units that weren't cut off from every supply; another failed two isolation checks only because its walks never reached those work steps.

What an attempt costs

Length

With only the request, the median attempt took 27 minutes and 49 tool calls (18 to 105 across the 7).

Cost

The median attempt cost $6.91 at list prices, from $1.38 to $14.97 across the 7; all 21 attempts came to $163. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the operator's and judges' calls aren't included.

What you license

With a licence

Revision 1 in all three versions: the hall's 12 files, the training and grading benches with the unit simulations, the 20 hidden faults, the grader and judges, a reference app graded in part, and the grader's report on each of the 21 attempts, as baseline results.

Format

The agent's container, the benches and a no‑network sandbox for the app. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, faults and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version.
• The reference app is graded only in part: its self‑test covers the app stage and 2 of the 20 faults.
• The attempts with only the request were graded earlier, with an earlier copy of some grader files, and partly by the grader's earlier operator: Sonnet 5.5 until 1 November, Opus 5.5 after; a Codex worker the maintainer authorized ruled some cases in the version with a hint.
• The operator is a model playing the electrician. A wrong reading pick is caught against the bench and the case collected again, but other slips, such as confirming a check it never made, can count against the app.
• A check fails when no graded walk reaches its step: in all 7 attempts with only the request, the stored‑charge wait on one unit revision's drive failed this way, not for an unsafe step.