Live server‑room cooling incident in a 3D building with model‑played staff

The brief gives the goal but no limits; each limit shows up in the world

Role

A facilities company's duty engineer on shift at a police station, acting only through one command, wa, in a 3D model of the station.

Request

Lead the incident: hold the server room at a safe temperature with the servers up until the cooling is restored, then report to the duty inspector and the facilities manager. Only people in the incident answer questions.

Deliverable

The world's state over five and a half hours of world time, read from its log, and a plain‑text incident report handed in through the same command, which ends the attempt.

What the reward depends on

A stopgap before calls and cause‑finding, as servers shut down on heat about 15 minutes in; the right people engaged early; safe use of circuits, sockets and the fire door; a report naming the cause, a lasting fix and owned actions, with nothing false.

Where the requirements are

In the world: panels, displays, labels and a circuit chart to inspect, and eight people who each hold part of the authority and the facts. The contractor admits cutting a cable only when asked, believing it the retired unit's own.

A 3D police station, a heat model and eight LLM‑played people

Interface

A shell in a fresh Docker container, with the agent's own CLI and one client, wa, that sends each command through one port to the 3D world in a headless browser outside it: move, look, inspect, use, ask, wait, screenshot and report.

Isolation

The world's port serves no files, so the rules, the people's briefs and the grader can't be read. The container could reach the internet; web search and fetch were off.

The building

Ten rooms, from the server room and control room to a store room, with doors, a fan, three portable air conditioners locked in a store, hoses, an extension reel and sockets. A heat model updates rack and room temperatures every world minute.

Simulated people

Eight, each played by Sonnet 5.5 from a brief of what they know, their limits and their tasks: five on site, reached in person or by radio, and three by phone. The world carries out an agreed task after its own duration.

Partial knowledge

Some briefs are wrong by design: the contractor believes rejoining the cable restarts the units, and the helpdesk believes only one manager can sign the form that releases the portable units.

Clock

World time moves only when the agent acts or waits: talking in person costs a minute, a phone call five, and replies come 1 to 10 world minutes later. The world is scored to five and a half hours and runs on after the report.

Limits

One session of up to 120 minutes, 150 turns and $20 at list prices.

31 checks of equal weight: 19 from the world's log, 12 on the report

Structure

31 checks in five families: keeping the servers cool (5), getting the right people involved (5), working safely (8), the report's cause and fixes (5), and the report as a true record (8).

World checks

Scripts score 19 checks from the log of temperatures, server states, circuit trips, people's actions and times; the task's audit removed two time‑above‑a‑limit checks.

Report checks

Two model judges, Opus 5.5 and GPT‑6.1 Sol, with no tools, read the report against a fact sheet built from the attempt's log. Both are among the models tested, and judged their own reports too.

Rulings

Verdicts where the two judges disagreed are ruled by hand against the log: 9 of 84 in the attempts with only the request.

Truth check

The false‑claims check fails any action, person or time the log contradicts; a promised time doesn't count as a record. It passes on a missing report.

Formula

Reward = checks passed / 31, each worth the same, with no gates. An attempt with no report fails 11 of the 12 report checks.

With the request or a hint, every model lost the servers to heat; given the method, six of seven kept them up

With only the request, every attempt let the servers overheat and shut down, and the best scored 77%. With a hint the mean went from 62% to 61%; given the manager's method step by step it rose to 81%, and Opus 5.5 met all 31 checks.

Each dot is one attempt; the line marks the median.

Each model made one attempt in each version. Fable 5.1's attempt told the rules was made again after a fault in how the people were played.

With a hint: the request plus a note from the facilities manager with nine pieces of advice, from starting a stopgap at once to restarting only after the contractor's job and permit are closed.

Told the rules: the request plus the manager's 13‑step method, including every part the report must contain.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request.

1. Servers lost to heat. In 7 of 7 attempts the agent made calls or chased the cause first, two never starting a stopgap at all, and the servers shut down on heat 15 to 20 minutes in.

2. False claims. All 6 reports handed in stated a fact or time the log contradicts; the seventh attempt handed in none.

3. No lasting fix. 5 of the 6 reports proposed no second control path, and 5 of 6 left open actions without an owner; the seventh attempt handed in no report.

4. Fire suppression left off. 4 of 7 attempts didn't get the gas fire suppression back in service; its panel shows a disconnected cylinder that no person's brief mentions.

5. Cooling not restored. In 3 of 7 the units never ran again on record: 2 handed in the report first, which ends the attempt, and 1 used all 150 turns.

What an attempt costs

Length

With only the request, the median attempt took 11 minutes of real time and 55 tool calls (18 to 190 across the 7).

Cost

$0.14 to $4.18 an attempt at list prices, median $1.76, for the agent model only; the model calls that play the people and the judges aren't included.

What you license

With a licence

Revision 1 in all three versions: the 3D world with its heat model and the wa client, the people's briefs, the grader with the judges' instructions and fact‑sheet builder, a scripted reference plan, and the grader's report on each of the 21 attempts, as baseline results.

Format

The agent runs in a Docker container; the 3D world runs in a headless browser outside it, reached through one port. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The world, briefs and grader aren't published.

What we haven't measured, and what's uneven

• Each model made one attempt in each version, so a model's score includes the variation between attempts.
• All 31 checks weigh the same: keeping the servers alive is three checks, as many as three phone calls.
• The two report judges are among the models tested and judged their own reports; verdicts where they disagreed were ruled by hand.
• In the self‑test, the scripted reference plan passes all 31 checks, with keyword matches standing in for 4 report checks and 8 passed by default. A plan that only calls the vendor, stops the works, tells the inspector and reports fully fails 9 of the 19 world checks.
• World time isn't real time: a long outage can happen in an attempt that takes a few real minutes.