AsapiYour AI facilities engineer
We built a police station in 3D and made the AI its facilities engineer, walking the building and talking to the people there, each played by an AI. In one job, a contractor has cut the cable that controls both of the server room's cooling units, and the AI must keep the servers running, then report. In the other, the server hall's temperature has swung all day since new cooling units went in, and it must find out why.
2 jobs · 7 AIs tested. Fable 5.1 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.
The two jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
Handle a server room's cooling failure (7 AIs tested)
On shift as a police station's duty engineer, the AI learns that a contractor has cut the cable that controls both server‑room cooling units, and has to keep the servers running until the units restart, then report what happened.
What the best AI did
- Found that one cable with no second path carried both units' controls, and was the only AI to propose adding a second one.
- Let the server room overheat until servers shut down.
How each AI scored
Find why a server hall's temperature swings (7 AIs tested)
Since a contractor replaced the two cooling units in a police station's server hall, its temperature has swung all day, and the AI, as the engineer on site, has to find the cause with tests it can undo and report it.
What the best AI did
- Found the cause, an open strip between each new unit and the wall, and explained why the old units had been steady.
- Didn't prove the cause with a test it could undo, such as covering the gap while a unit ran.
How each AI scored
Who did best
Anthropic's Fable 5.1 did best on both jobs, and still failed both: like every AI, it let the servers overheat when the cooling stopped. Its reports were the only ones to explain each cause in full, and it was the only AI to propose a second path for the units' controls. OpenAI's GPT‑6 Astra came second on average, and GPT‑6 Luna, a small AI, came last.
Average score out of 100 over the jobs each AI was tested on, for the 7 AIs tested on both jobs.
What the two jobs showed
What they do well
They stop the damage, and leave things as they found them.
During the cooling failure, all seven AIs stopped the contractor's work before it did more damage. In the server hall, all seven put back every setpoint, floor tile and cover they had touched.
They tell most of the people who need to know.
When the cooling stopped, all seven told the IT service owner and the control room supervisor within 30 minutes, and six logged the incident with the facilities helpdesk. Two were late telling the duty inspector.
Where they fall short
They couldn't keep the servers cool once the cooling stopped.
All seven let the racks pass 35 °C until servers shut down on heat, and the dispatch and records servers were down for more than 30 minutes in total. Given the manager's method step by step, Opus 5.5 kept them running and got everything right.
Their reports claim more than they showed.
All six reports handed in on the cooling failure included at least one false claim, and five of the seven on the server hall. There, every AI also missed the test that proves the cause: covering the gap behind a unit while it runs and watching the swings settle.
How these jobs were set up and scored
- The building
- A 3D model of a police station. The AI moves around it, reads panels and displays, uses things like doors, fans and portable cooling units and talks to people, all by typing one command. Time in the building moves only when the AI acts or waits. Each person is played by Sonnet 5.5: eight in the cooling failure, and one IT technician in the server hall.
- Scores
- Each score is the share of the job's checks met, all of equal weight, and 100 means everything right: 31 checks on the cooling failure and 20 on the server hall. Scripts check the building's log, such as temperatures and which servers ran, and two AIs, Opus 5.5 and GPT‑6.1 Sol, read each report, their own included, with a person settling any split.
- Versions of the request
- The scores here are for the request alone. We also ran each job with a short note of advice from the manager, and with the manager's method step by step. With the method, five of the seven AIs scored 94 or more on the cooling failure, and four scored 95 on the server hall.
- Time and cost
- Each AI had one try at each job, within 120 minutes, 150 turns and $20. The tries took 1 to 18 minutes of real time and cost from 14 cents to $4.18 at published prices.
Every AI's score on every job
| AI | Handle a server room's cooling failure | Find why a server hall's temperature swings |
|---|---|---|
| Fable 5.1Anthropic | 77 | 70 |
| GPT‑6 AstraOpenAI | 65 | 65 |
| GPT‑6 LunaOpenAI | 58 | 30 |
| GPT‑6.1 SolOpenAI | 68 | 55 |
| Opus 5.5Anthropic | 74 | 45 |
| Sonnet 5Anthropic | 35 | 60 |
| Sonnet 5.5Anthropic | 58 | 40 |
These are the scores out of 100 for the request as a person would write it, one try per AI. Darker means higher.