AsapiThe AI Agent Trials
Asapi / Catalogue / Watchdog program commissioned for a running solvent recovery unit
Environment datasheet · Process control

Watchdog program commissioned for a running solvent recovery unit

ID tb‑tasks/pid‑guardian · revision 1 · results as of 9 November 2026

As commissioning engineer for a company selling watchdog programs to small plants, the agent rehearses on a copy of the live plant and leaves a program on the gateway that must hold the unit inside its limits, hold back on harmless events and escalate only when it must.

Checks
41, 1 point each; the task's own pass needs all
Tested
6 models, one attempt each
Best
93% (GPT‑6 Astra)
Median
79%
Full solves
0 of 6
Median attempt
28 min, 56 tool calls
What it tests

A watchdog left in charge of a running plant, through a hidden day

Long‑horizon work

After its session ends, the agent's program runs the plant alone for a day, through a reboot, keeping only its own files.

Stateful tool use

Writes land on a running plant at once, so the agent can rehearse on a copy first, and must stay within its agreed scope.

Agentic coding

The watchdog must start from the gateway's scheduler, survive a reboot that kills every process, and keep its state.

When to act or escalate

The watchdog must stop the right pump in a leak and escalate only inside the leak window, without shutting down on a harmless afternoon.

Catching errors

Odd records for equipment that works fine must not be reported as defects, and a one‑point temperature spike must not shut anything down.

How it works

A two‑train plant simulator, a rehearsal copy and a gateway that reboots

The agent's world

A shell in a Linux container with a client for the plant service and the gateway's file system; no network.

Rehearsal

A sandbox copy of the live plant where time is free and faults can be injected, from pump trips to wrong field reports.

The graded day

After the session, the plant runs on for about a day under the program, through a reboot, with disturbances it never saw.

The reward

41 one‑point checks on the plant's logs and the deliverables, decided by script; the task's own pass needs all 41.

Results

Five attempts scored 71–93%, and one that saved no program 20%

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each79%0 of 6
ModelReward
GPT‑6 AstraOpenAI93%
GPT‑6.1 SolOpenAI88%
Sonnet 5.5Anthropic80%
Fable 5.1Anthropic78%
GPT‑6 LunaOpenAI71%
Opus 5.5Anthropic20%
What models get wrong

They stop the wrong pump, act on harmless blips, and miss an escalation

When to act or escalate

  1. Of the five that deployed a program, four left a leaking pump running when a field report gave the wrong tag.

  2. 33% of attempts cut a heater on a one‑point temperature spike.

  3. Two of the five programs never escalated a leak that couldn't be shut off, treating its report as information.

Long‑horizon work

  1. One stopped acting once it had escalated, and a tank ran under its low limit through the morning.

Stateful tool use

  1. One never rehearsed on the plant and wrote its program from the documents, keeping a swapped pair of points as indexed.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.03 to $15.80 for the agent model; grading is a script, with no model calls.

What's in it

The plant simulator and service, the gateway, the documents and drawings, the hidden day and the grader, and a reference watchdog that passes.

Format

Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the plant service beside the agent.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.