Scanned P&ID set digitised into instrument, line and equipment lists
ID tb‑tasks/pid‑digitise · version 4.0.1 · tested 31 October 2026 on 4.0.0, regraded on 4.0.1
As a digitisation consultant on a fixed price, the agent reads a plant unit's ten scanned sheets into an index of about 400 instruments, a list of about 230 lines and about 100 equipment items, and checks the control system's out‑of‑date export against them.
- Checks
- 3,836, 12 gates; each invented row adds a failed check
- Tested
- 5 models, one attempt each
- Best
- 100% (Sonnet 5.5)
- Median
- 99%
- Full solves
- 1 of 5
- Median attempt
- 17 min
About 730 plant items read off scanned drawings, at a fixed price
Multimodal records
Every row is read from ten scanned sheets, some faded or hand‑marked, where no tag is readable at whole‑sheet size.
Clarifying with people
A scripted query desk answers only what is asked, and the client's practice on control‑system points is learnable only from it.
When to act or escalate
Desk checks are charged against the fixed price, so checking everything costs more than the job pays, and a guessed digit fails.
Catching errors
The control system's out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported.
Unstated requirements
The client's practice on which instruments get a control‑system point is found only through the priced query desk.
Ten scanned sheets, a priced query desk and no OCR
The agent's world
A shell in a Linux container with Python and image libraries but no OCR; scans are read by cropping and enlarging regions.
The scans
A legend and nine drawings of 9,933 by 7,016 pixels, for two process trains, cooling water and a flare drum, some faded or hand‑marked.
Limits
One session of 90 minutes, against an expert's estimate of six hours.
The reward
One point per row and field across the three lists, eight reconciliation rows and a 10‑point price check; each invented row adds a failed check.
Four attempts scored 99% or more, and one stopped early at 45%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Sonnet 5.5Anthropic | 100% |
| Opus 5.5Anthropic | 99% |
| GPT‑6.1 SolOpenAI | 99% |
| GPT‑6 AstraOpenAI | 99% |
| GPT‑6 LunaOpenAI | 45% |
Three of the four also passed the task's own bar; the fourth delivered a line that doesn't exist.
They connect gauges to the nearest pipe, misread a digit unchecked, and trust the export
Multimodal records
60% put a field gauge's connection on the nearest pipe or the wrong line, on rows no desk check covered.
Two numbered a dense pump group's branch lines in the wrong order, putting line ends and gauges on the wrong lines.
When to act or escalate
One misread a digit in a line number and delivered a line that doesn't exist, an error the engagement letter rejects outright, without checking that service at the desk.
Catching errors
One built the index from the out‑of‑date export, missing 161 instruments, among them all 23 emergency‑shutdown level switches, and stopped after 4 minutes.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.03 to $22.05 for the agent model; grading is a script, with no model calls.
What's in it
The scans and legend, the engagement letter and export, the query desk, the grader and answer key, and a passing reference solution.
Format
A Harbor task in a Linux container, with the desk as a root‑owned script.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The sketched‑drawings environment uses the same plant, desk, export, price and grader, with pencil sketches instead of scans. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials