Watchdog program commissioned for a running solvent recovery unit
ID tb‑tasks/pid‑guardian · revision 1 · results as of 9 November 2026
As commissioning engineer for a company selling watchdog programs to small plants, the agent rehearses on a copy of the live plant and leaves a program on the gateway that must hold the unit inside its limits, hold back on harmless events and escalate only when it must.
- Checks
- 41, 1 point each; the task's own pass needs all
- Tested
- 6 models, one attempt each
- Best
- 93% (GPT‑6 Astra)
- Median
- 79%
- Full solves
- 0 of 6
- Median attempt
- 28 min, 56 tool calls
A watchdog left in charge of a running plant, through a hidden day
Long‑horizon work
After its session ends, the agent's program runs the plant alone for a day, through a reboot, keeping only its own files.
Stateful tool use
Writes land on a running plant at once, so the agent can rehearse on a copy first, and must stay within its agreed scope.
Agentic coding
The watchdog must start from the gateway's scheduler, survive a reboot that kills every process, and keep its state.
When to act or escalate
The watchdog must stop the right pump in a leak and escalate only inside the leak window, without shutting down on a harmless afternoon.
Catching errors
Odd records for equipment that works fine must not be reported as defects, and a one‑point temperature spike must not shut anything down.
A two‑train plant simulator, a rehearsal copy and a gateway that reboots
The agent's world
A shell in a Linux container with a client for the plant service and the gateway's file system; no network.
Rehearsal
A sandbox copy of the live plant where time is free and faults can be injected, from pump trips to wrong field reports.
The graded day
After the session, the plant runs on for about a day under the program, through a reboot, with disturbances it never saw.
The reward
41 one‑point checks on the plant's logs and the deliverables, decided by script; the task's own pass needs all 41.
Five attempts scored 71–93%, and one that saved no program 20%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 93% |
| GPT‑6.1 SolOpenAI | 88% |
| Sonnet 5.5Anthropic | 80% |
| Fable 5.1Anthropic | 78% |
| GPT‑6 LunaOpenAI | 71% |
| Opus 5.5Anthropic | 20% |
They stop the wrong pump, act on harmless blips, and miss an escalation
When to act or escalate
Of the five that deployed a program, four left a leaking pump running when a field report gave the wrong tag.
33% of attempts cut a heater on a one‑point temperature spike.
Two of the five programs never escalated a leak that couldn't be shut off, treating its report as information.
Long‑horizon work
One stopped acting once it had escalated, and a tank ran under its low limit through the morning.
Stateful tool use
One never rehearsed on the plant and wrote its program from the documents, keeping a swapped pair of points as indexed.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.03 to $15.80 for the agent model; grading is a script, with no model calls.
What's in it
The plant simulator and service, the gateway, the documents and drawings, the hidden day and the grader, and a reference watchdog that passes.
Format
Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the plant service beside the agent.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials