AsapiThe AI Agent Trials
Asapi / Catalogue / Offline fault‑finding tablet app for data‑centre cooling units
Environment datasheet · Facilities

Offline fault‑finding tablet app for data‑centre cooling units

ID laycode/task11 · revision 1 · results as of 2 November 2026

A data centre's facilities lead wants a tablet app that leads a new night‑shift electrician, by touch alone, from a cooling unit that's down to its root cause, the repair and a filled‑in report. The agent builds it from the hall's records and a training bench.

Checks
79, 40 points on faults, 60 on the app
Tested
7 models, one attempt each
Best
87% (Opus 5.5)
Median
64%
Full solves
0 of 7
Median attempt
27 min, 49 tool calls
What it tests

A tablet app that leads a new electrician from a dead unit to its root cause

Agentic coding

The offline app is walked through 20 faults it never saw, and a fix counts only if the unit then runs cleanly.

Multimodal records

The drawings are PDFs read only by looking, and each test point the app names must match that unit's drawing.

Catching errors

A blown fuse usually hides a shorted coil or winding, and some causes show only in the alarm history or maintenance log.

Unstated requirements

Only 3 of 79 checks cite the request; the rest include basics such as cutting the power before any work on a unit.

How it works

12 files, a training bench of 12 units, and 20 hidden faults

The agent's world

A fresh container with the hall's 12 files, among them drawings readable only by looking, worked through the model's usual coding agent.

The bench

A training bench simulates each of the 12 units from its wiring, with meters, displays, alarms and 8 practice faults; it refuses the graded ones.

Grading

An operator model plays the electrician, one fresh session per hidden fault, told only the unit and that it's down; the app runs offline.

The reward

Each of 20 faults earns two points, fixed and its root cause named; 39 app checks share 60, from safe isolation to the report.

Results

Five of the seven scored 47–68%; the best reached 87% and the lowest 38%

Each dot is one attempt, and the line marks the median.

MedianFully solved
7 modelsOne attempt each64%0 of 7
ModelReward
Opus 5.5Anthropic87%
GPT‑6 AstraOpenAI68%
GPT‑6.1 SolOpenAI66%
Sonnet 5.5Anthropic64%
Fable 5.1Anthropic64%
Sonnet 5Anthropic47%
GPT‑6 LunaOpenAI38%

With an IT specialist's 23 requirements added to the request, the mean rose from 62% to 72%, and none fully solved it.

What models get wrong

They miss a loose wire, stop at the blown fuse, and leave causes unnamed

Catching errors

  1. Every attempt missed a loose motor lead at one unit's drive output, neither fixing nor naming it.

  2. Every attempt left the shorts behind the chilled‑water units' blown fuses unfixed: apps kept fitting new fuses, gave no next step, or left the fuse open.

  3. Every attempt left at least 5 of the 20 root causes unnamed, and one app's reports gave only symptoms or lists of possible causes.

Agentic coding

  1. One app's report screen could never be reached, so its walks named no root cause at all.

Unstated requirements

  1. One attempt had resistance measured on units that weren't cut off from every supply.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $6.90 at list prices for the agent model; the operator's and judges' calls are extra.

What's in it

The hall's 12 files, the training and grading benches, the 20 hidden faults, all three versions, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the agent's container, the benches and an offline sandbox for the app run side by side.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.