Server‑hall cooling diagnosis by undoable tests in a 3D building

Nothing is broken; the cause hides behind the units, among plausible suspects

Role

A facilities company's on‑site engineer at a police station, acting only through one command, wa, in a 3D model of the building.

Request

Work out what causes the swings without repairing anything or disturbing the station's work, leave the hall as found, and report the cause, its evidence and a remedy to the owner. The request names no method.

Deliverable

The world's state over the test and at the end, and the report, handed in through the same command, which ends the attempt.

What the reward depends on

Cooling kept, no unit moved and the exit shut; every setpoint, cover, floor tile and the duty unit put back; the cause proven by a cover test on each unit; a report that names it, rules out the tuning and the sensor with evidence, and claims nothing false.

Where the requirements are

In the world: two units with controllers, a trend screen, probes, the contractor's handover binder and tools in a cupboard. The cause, a 5 cm open strip behind each unit, shows only to a torch from above or under a lifted floor tile.

A 3D police station with a heat and control model and one technician

Interface

A shell in a fresh Docker container, with the agent's own CLI and one client, wa, that sends each command through one port to the 3D world in a headless browser outside it: move, look, inspect, use, ask, wait, screenshot and report.

Isolation

The world's port serves no files, so the rules, the technician's brief and the grader can't be read. The units' tuning values are locked. The container could reach the internet; web search and fetch were off.

The building

Two cooling units, duty and standby, in the hall, with controllers, a valve dial and rack inlet probes; a trend screen in the IT office and the chilled‑water display in the plant room; an air logger, a torch, a tile lifter, a board and a card in the facilities cupboard.

Suspects

Plausible but wrong: the locked tuning, the sensor, the valve, chilled water and the setpoint. Real but small: open rack spaces with no blanking panels and a misplaced perforated floor tile.

Simulated person

One IT technician, played by Sonnet 5.5 and reached only in person. She knows four things she saw, says she doesn't know the cause, and can change nothing.

Clock

World time moves only when the agent acts or waits. The hall swings on a cycle of about 7 minutes, and a unit just put on duty swings for 10 to 20 minutes before it settles.

Limits

One session of up to 120 minutes, 150 turns and $20 at list prices; 4 of the 21 attempts, none with only the request, used all 150 turns.

20 checks of equal weight: 8 from the world, 12 on the report

Structure

20 checks in five families: working safely (2), leaving the hall as found (4), proving the cause by test (5), the report's cause (6), and its evidence, fix and truth (3).

World checks

Scripts score 8 checks from the log and the final state. A cover test passes when the swing settles while that unit alone carries the hall with its strip covered.

Report checks

Two model judges, Opus 5.5 and GPT‑6.1 Sol, with no tools, read the report against a fact sheet holding the hidden truth and everything the engineer did, saw and measured. Both are among the models tested.

Rulings

Verdicts where the two judges disagreed are ruled by hand: 10 of 84 in the attempts with only the request.

Formula

Reward = checks passed / 20, each worth the same, with no gates. An attempt with no report fails all 12 report checks and the four put‑back checks.

With only the request, two models named the cause and none proved it by test; given the method, four proved it on both units

With only the request, every attempt kept the hall safe and put everything back, and none ran a cover test on either unit; the best scored 70%. With a hint the mean went from 52% to 50%, and given the method, to 64%.

Each dot is one attempt; the line marks the median.

Each model made one attempt in each version. Six Codex attempts that stopped within 20 seconds on a test‑setup fault were made again.

With a hint: the request plus a note from the facilities manager with seven pieces of advice, among them to prove a cause with an undoable test on every unit.

Told the rules: the request plus the manager's 9‑step method, including every part the report must contain.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request.

1. No proof by test. 7 of 7 attempts ran no cover test on either unit, so no report could cite one; 2 found the gap and handed in within 11 world minutes without testing.

2. The sensor not ruled out. 7 of 7 reports gave no evidence, such as a logger held beside it, that the return sensor wasn't at fault.

3. The cause not named. 5 of 7 reports didn't name the open strip behind each unit, and 5 didn't say why the old units were steady or tie the swings to the replacement.

4. Claims beyond the record. 5 of 7 reports claimed something the log doesn't support, such as readings never taken or both units running when one stood by.

What an attempt costs

Length

With only the request, the median attempt took 6 minutes of real time and 57 tool calls (29 to 123 across the 7).

Cost

$0.20 to $2.67 an attempt at list prices, median $0.89, for the agent model only; the model calls that play the technician and the judges aren't included.

What you license

With a licence

Revision 1 in all three versions: the 3D world with its heat and control model and the wa client, the technician's brief, the grader with the judges' instructions and fact‑sheet builder, a scripted reference walk‑through, and the grader's report on each of the 21 attempts, as baseline results.

Format

The agent runs in a Docker container; the 3D world runs in a headless browser outside it, reached through one port. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The world, brief and grader aren't published.

What we haven't measured, and what's uneven

• Each model made one attempt in each version.
• Handing in no report scores 10%, as four of the 21 attempts did; doing nothing but write a one‑line report would score about 30 to 40%.
• The two report judges are among the models tested and judged their own reports; verdicts where they disagreed were ruled by hand.
• The scripted reference walk‑through has been shown to pass all 20 checks only in the self‑test, with keyword stand‑ins for the judges; it hasn't been graded by the real judges.
• Only the true cause passes a cover test: covering the grille, the sensor or the pipes changes nothing, so a lucky guess can't pass the test checks.