Scanned P&ID set digitised into instrument, line and equipment lists

Every tag read from enlarged crops, the doubts worth paying for checked at the desk

Role

A digitisation consultant engaged at a fixed price, with no CAD source, only scans.

Request

Digitise the scans into three lists, of instruments, lines and equipment, as built, and reconcile the instrument list with the plant's control‑system tag export, within the fixed price and the acceptance rules.

Deliverable

Four CSV files with exactly the columns the instruction lists.

What the reward depends on

Every item found and every field read right, nothing invented, the eight discrepancies with the control system found, and charges within the fixed price.

Where the requirements are

In ten scanned A1 sheets, some faded, some with red‑line mark‑ups and smudged tags, the engagement letter, which lists the errors that reject a delivery outright, and the client's practice, known only at the desk.

Ten scans at 300 dpi and a desk that charges for every answer

Interface

A shell in a Linux container with Python, Pillow and numpy and no OCR package; the scans are read by cropping and enlarging regions. Codex ran with web search off; Claude Code's web tools were on for the two Claude models, and neither used them.

Scans

Ten sheets of 9,933 by 7,016 pixels: a legend and nine piping and instrumentation drawings, for two process trains, cooling water and a flare drum.

Query desk

Fixed code answers checks on a tag, line or equipment item, discrepancy checks and site visits to read an instrument's nameplate, from the client's as‑built record, each charged against the fixed price.

Limits

One session of 90 agent minutes, against an expert's estimate of six hours.

3,836 checks by script, with gates that reject a delivery outright

Structure

One point per key row and field over the three lists, 8 reconciliation rows, a 10‑point price check and 12 zero‑point gates, plus a failed check for each invented or extra row.

The task's pass

The task's own pass, kept beside the reward, needs set rates of rows found, rows invented and fields right in every list, the reconciliation exactly right, charges within the price and no critical error, such as an invented row or a wrong status.

Who decides

The grader matches each delivered row to the as‑built answer key, checks every field, counts invented rows and adds up the desk's ledger; script only. Version 4.0.1 widened two key entries and corrected one desk answer, and every attempt was regraded.

Formula

Reward = points earned / points available, from 0 to 1; the task's own pass needs all 12 gates. Writing nothing scores 0, with 430 critical errors.

Three models passed; one lost the pass to a single misread digit

Sonnet 5.5 got every row right; Opus 5.5, one field wrong, and GPT‑6 Astra also passed every gate. GPT‑6.1 Sol scored 99.6% but delivered a line that doesn't exist and failed. GPT‑6 Luna stopped early at 45%.

Each dot is one attempt; the line marks the median.

One attempt per model, five models.

Sonnet 5.5 passes under version 4.0.1; as first graded, one wrong status had failed it.

What the models got wrong

Counted over the 5 models, one attempt each.

1. One digit, one rejection. 1 of 5 misread a digit in a line number and delivered a line that doesn't exist, an error the engagement letter rejects outright, and never checked a line of that service at the desk.

2. Branches in the wrong order. 2 of 5 numbered the branch lines of a dense pump group in the wrong order, putting line ends and gauges on the wrong lines.

3. Connections to the nearest pipe. 3 of 5 put a field gauge's process connection on the nearest pipe or the wrong line, on rows no desk check covered.

4. The export taken for the drawings. 1 of 5 built the index from the out‑of‑date export, leaving out 161 instruments, among them all 23 level switches on the emergency‑shutdown system, and stopped after 4 of its 90 minutes.

What an attempt costs

Length

The median attempt took 17 minutes and 35 model calls (25 to 331 across the 5); tool calls weren't recorded for four of the five.

Cost

The median attempt cost $4.67, from $0.03 to $22.05 across the 5. All five are estimates from token counts, since every attempt ran on a plan sign‑in not billed per call; GPT‑6 Luna's also uses a derived price, as it has none published.

What you license

With a licence

Version 4.0.1: the scans and legend, the engagement letter and export, the query desk over the as‑built record, the grader and answer key, a reference solution that passes, and the grader's report on each of the 5 attempts, as baseline results.

Format

A Harbor task in a Linux container, with the desk as a root‑owned script.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The scans, key and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model.
• Version 4.0.1 hasn't been run again in Harbor; only the reference solution was regraded, locally.
• The score and the pass can disagree: 99.6% failed on one critical error, while 99.5% passed.