Watchdog program commissioned for a running solvent recovery unit

A program left in charge of a plant for a day, without its author

Role

The commissioning engineer for a company whose watchdog program stands in for a night control room at small plants, handed a running two‑train unit in the morning.

Request

Map the plant's data points to the drawings, list where the documents are wrong, and leave a watchdog in charge on the site's gateway that afternoon, started by its scheduler and surviving its daily reboot.

Deliverable

The watchdog program in its folder, plus a map of every data point to the instrument it really carries and a defect list giving item, problem and evidence.

What the reward depends on

What the plant does under the watchdog through a hidden day: limits held, nothing shut down without need, escalation only when needed, every action in scope, reports handled as the manual says; plus the map and defects.

Where the requirements are

In the brief's judging list, a scope agreement, operating manual, limits table and data sheets, which the brief warns aren't all accurate, drawings that are, the live plant, and the gateway's own files.

A live plant, a rehearsal copy, and a gateway that reboots

Interface

A shell in a Linux container with Python and ezdxf, a command‑line client for the plant service (read, history, write, mode, field requests, escalate, advance, sandbox) and the gateway's file system.

The plant

A simulator of two parallel trains with shared tanks and utilities, scanned every 5 seconds of plant time. Its clock moves only when the agent advances it, and the live plant stops at the handover.

Rehearsal

A sandbox copy of the live plant, where time is free and faults can be injected: pump trips, cooling‑water loss, frozen or glitched points, stuck valves and field reports.

The gateway

A simulated box whose scheduler runs in plant time. A reboot kills every process and clears everything but the watchdog's own folder.

Simulated people

Scripted, with no live model: a field operator on each shift who carries out requests within set limits, and four people whose free‑text reports, some with wrong tags or kinds, are posted to the watchdog.

Network

None. Codex ran with web search off; Claude Code's web tools were on for the three Claude models, and no attempt used them.

Limits

One session of 120 agent minutes, against an expert's estimate of eight hours; attempts used 9 to 53 minutes.

41 checks read from a hidden day's plant logs

Structure

41 one‑point checks in seven families: forms and map (4), defects (8), safe limits (8), production and no needless shutdown (4), escalation and scope (7), the watchdog running and its inbox (4), reports and state across the reboot (6).

The graded day

After the session the verifier stops every process of the agent and runs the plant on for about a day under whatever was deployed, through the reboot, with disturbances and reports the agent never saw.

Who decides

The grader reads the plant's logs for the graded day and the three deliverables; the defect list is free text, graded on the item. Script only, no LLM judge.

The task's pass

The task's own pass, kept beside the reward, needs all 41 checks.

Formula

Reward = checks passed / 41, from 0 to 1; the task's own pass needs all 41. Doing nothing passes 5, since the checks that reward holding back count only when a watchdog was deployed.

Self‑test

The reference watchdog passes all 41 in Harbor. Grader revisions 1.0.1 and 1.0.2 came after the attempts, and every attempt was regraded offline from its own plant record.

No model passed; the best missed 3 of 41 checks

GPT‑6 Astra met 38 of 41 checks (93%), GPT‑6.1 Sol 36 (88%), Sonnet 5.5 33 (80%), Fable 5.1 32 (78%) and GPT‑6 Luna 29 (71%). Opus 5.5 saved no program and scored 20% on plant outcomes alone.

Each dot is one attempt; the line marks the median.

One attempt per model, six models, regraded under grader 1.0.2. Only Sonnet 5.5 crossed no limit all day.

Opus 5.5's attempt ended when Claude Code's safety classifier stopped the turn that wrote its program; the model then ended the session, and the attempt counts.

What the models got wrong

Counted over the 6 models, one attempt each.

1. The leaking pump left running. 5 of 6 didn't stop a leaking pump reported under the wrong tag: two took the tag as the pump, one only logged it, one started the spare but never stopped the leaking pump, one had no program. Only GPT‑6.1 Sol read the report's words.

2. After the leak. 1 of 6 stopped acting once it had escalated, and a tank ran under its low limit through the morning; 2 of 6 kept a train off feed against the day crew restoring the unit, so it was still off at the end.

3. Harmless events acted on. 2 of 6 cut a heater on a one‑point temperature spike, and 1 of 6, reading reports by keywords, put both trains' reflux loops in manual over an irrelevant report.

4. An escalation missed. 2 of the 5 that deployed a program never escalated a leak nobody could isolate: one filed the report as information because of the kind its sender chose, the other read it as the day crew's information.

5. Documents trusted. 1 of 6 never rehearsed on the plant and wrote its program from the documents; its map kept a swapped pair as indexed, and it found 3 of the 7 required defects.

6. No program saved. 1 of 6 had the turn that wrote its program stopped by Claude Code's safety classifier, then ended the session without the program, the map or the defect list, passing 8 of 41 on plant outcomes alone.

What an attempt costs

Length

The median attempt took 28 minutes and about 56 tool calls (18 to 133 across the 6).

Cost

Claude Code's own figures for its three attempts: a median of $5.40, from $2.74 to $15.80. The three Codex attempts, estimated at list prices from their tokens: a median of $1.41, from $0.03 to $7.85. None is a bill: every attempt ran on a plan sign‑in.

What you license

With a licence

Revision 1: the plant simulator and service, the gateway, the documents and drawings, the hidden day and the grader, a reference watchdog that passes, and the grader's report on each of the 6 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor, in a Linux container, with the plant service running as root beside the agent.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The simulator, records and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• One plant and one hidden day, with one escalation, one pump trip and one mistagged report, so a single event can decide a check.
• Opus 5.5's 20% follows a stop by Claude Code's safety classifier: it had found most of the defects the others found, but wrote none of them down.
• The checks that reward holding back pass only when a watchdog was deployed, so doing nothing earns 5 of 41.