Photographed P&ID sketches digitised into instrument, line and equipment lists
ID tb‑tasks/pid‑digitise‑sketch · revision 1 · results as of 10 November 2026
As a digitisation consultant on a fixed price, the agent reads a plant unit's pencil sketches, seven sheets split over two pages, into an index of about 400 instruments, a list of about 230 lines and about 100 equipment items, and checks the control system's out‑of‑date export against them.
- Checks
- 3,836, 12 gates; each invented row adds a failed check
- Tested
- 5 models, one attempt each
- Best
- 99% (GPT‑6 Astra)
- Median
- 99%
- Full solves
- 0 of 5
- Median attempt
- 18 min
About 730 plant items read off pencil sketches, at a fixed price
Multimodal records
Every row is read from 17 photographed pencil pages in the drafter's own symbols, with look‑alike digits to tell apart.
Clarifying with people
A scripted query desk answers only what is asked, and the client's practice on control‑system points is learnable only from it.
When to act or escalate
Desk checks are charged against the fixed price, so checking everything costs more than the job pays, and a guessed digit fails.
Catching errors
The control system's out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported.
Unstated requirements
The client's practice on which instruments get a control‑system point is found only through the priced query desk.
17 photographed pages, a priced query desk and no OCR
The agent's world
A shell in a Linux container with Python and image libraries but no OCR; pages are read by cropping and enlarging regions.
The drawings
A legend and nine sheets, seven split over two pages at a match line, with smudged tags, site clutter and labels crossed by pipes.
Limits
One session of 90 minutes, against an expert's estimate of six hours.
The reward
One point per row and field across the three lists, eight reconciliation rows and a 10‑point price check; each invented row adds a failed check.
Four attempts scored 99%, and one stopped early at 39%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 99% |
| Opus 5.5Anthropic | 99% |
| Sonnet 5.5Anthropic | 99% |
| GPT‑6.1 SolOpenAI | 99% |
| GPT‑6 LunaOpenAI | 39% |
Three of the four also passed the task's own bar; the fourth delivered three rows that don't exist.
They misplace pump‑group connections, misread labels unchecked, and trust the export
Multimodal records
Every attempt that read the drawings put a gauge's connection or a line's end on the wrong branch of a pump group, on rows never checked at the desk.
When to act or escalate
One misread three labels and delivered three rows that don't exist, errors the engagement letter rejects outright, without checking any at the desk.
Catching errors
One built the index from the out‑of‑date export, missing 168 instruments, 29 of them safety instruments, and stopped after 10.5 minutes.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.05 to $29.04 for the agent model; grading is a script, with no model calls.
What's in it
The photographed pages and legend, the engagement letter and export, the query desk, the grader and answer key, and a passing reference solution.
Format
Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the desk as a root‑owned script.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The plant, desk, export, price and grader match the scanned‑drawings environment, so the two differ only in the drawings. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials