Photographed P&ID sketches digitised into instrument, line and equipment lists

Every tag read from pencil, the doubts worth paying for checked at the desk

Role

A digitisation consultant engaged at a fixed price by a plant whose only as‑built drawings are pencil sketches, photographed.

Request

Digitise the sketches into three lists, of instruments, lines and equipment, as built, and reconcile the instrument list with the plant's control‑system tag export, within the fixed price and the acceptance rules.

Deliverable

Four CSV files with exactly the columns the instruction lists.

What the reward depends on

Every item found and every field read right, nothing invented, the eight discrepancies with the control system found, and charges within the fixed price.

Where the requirements are

In 17 photographed pages, the engagement letter, which lists the errors that reject a delivery outright, and the client's practice, known only at the desk.

Seventeen pencil pages and a desk that charges for every answer

Interface

A shell in a Linux container with Python, Pillow and numpy and no OCR package; the pages are read by cropping and enlarging regions. Codex ran with web search off; Claude Code's web tools were on for the two Claude models, and neither used them.

Drawings

A legend and nine sheets, seven drawn on two pages joined at a match line, 3,730 to 9,933 by 7,016 pixels: the drafter's own symbols and hand, coloured‑pencil site clutter, smudged tags, and labels crossed by pipes.

Query desk

Fixed code answers checks on a tag, line or equipment item, discrepancy checks and site visits to read an instrument's nameplate, from the client's as‑built record, each charged against the fixed price.

The same plant

Plant, answer key, desk, export, price and grader are those of the scanned‑drawings environment; only the drawings differ, so its results are a control for reading hand drawings.

Helpers

Both Claude Code attempts split the sheets among helper agents; the Codex attempts worked alone.

Limits

One session of 90 agent minutes, against an expert's estimate of six hours.

3,836 checks by script, with gates that reject a delivery outright

Structure

One point per key row and field over the three lists, 8 reconciliation rows, a 10‑point price check and 12 zero‑point gates, plus a failed check for each invented or extra row.

The task's pass

The task's own pass, kept beside the reward, needs set rates of rows found, rows invented and fields right in every list, the reconciliation exactly right, charges within the price and no critical error, such as an invented row or a wrong status.

Who decides

The grader matches each delivered row to the as‑built answer key, checks every field, counts invented rows and adds up the desk's ledger; script only. Key revisions 1.0.1 and 1.0.2 accept second readings of one connection and five more, and every attempt was regraded.

Formula

Reward = points earned / points available, from 0 to 1; the task's own pass needs all 12 gates. Writing nothing scores 0, with 430 critical errors.

Three models passed; one lost the pass to three invented rows

GPT‑6 Astra, Opus 5.5 and Sonnet 5.5 passed every gate, scoring 99% after losing 3, 6 and 17 points. GPT‑6.1 Sol also scored 99%, but delivered three rows that don't exist and failed. GPT‑6 Luna stopped early at 39%.

Each dot is one attempt; the line marks the median.

One attempt per model, five models; Fable 5.1 hasn't been tested on it yet.

On the scanned drawings, with the same plant, desk and grader, the same three models passed and the same two failed.

Every attempt kept its charges within the fixed price.

What the models got wrong

Counted over the 5 models, one attempt each.

1. Pump groups left unchecked. 4 of 5, all those that read the drawings, put a gauge's connection or a line's end on the wrong branch of a pump group, on rows never checked at the desk; these were most of the 3 to 17 points the passing attempts lost.

2. Three misread labels. 1 of 5 misread a level glass's tag, a line's size and a pump discharge's number, delivering three rows that don't exist, errors the engagement letter rejects outright. It checked none of them at the desk; it scored 99% and failed.

3. The export taken for the drawings. 1 of 5 built the index from the out‑of‑date export, missing 168 instruments, 29 of them safety instruments, wrote only the 20 vent lines of 231, spent most of the price on desk checks of those, and stopped after 10.5 minutes.

4. Weak rows named, not checked. 1 of 5 said in its closing note that about half its line ends rested on the numbering pattern alone, and left them unqueried; its 17 lost points were gauge connections and line ends.

What an attempt costs

Length

The median attempt took 18 minutes and 40 model calls (35 to 358 across the 5); tool calls weren't recorded for four of the five.

Cost

Claude Code's own figures for its two attempts: $12.51 and $29.04. The three Codex attempts, Harbor's estimates from their tokens: a median of $1.05, from $0.05 to $5.72. None is a bill: every attempt ran on a plan sign‑in.

What you license

With a licence

Revision 1: the photographed pages and legend, the engagement letter and export, the query desk over the as‑built record, the grader and answer key, a reference solution that passes, and the grader's report on each of the 5 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor, in a Linux container, with the desk as a root‑owned script.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The sheets, key and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model.
• The answer key changed twice on 10 November, accepting second readings of one connection and then of five more; the first change came before the three Codex attempts. Every attempt was regraded under the final key, and no pass changed.
• The score and the pass can disagree: GPT‑6.1 Sol's 99% failed on three invented rows, while Sonnet 5.5's 99% passed.
• Opus 5.5's recorded 71 minutes include about 32 minutes waiting on its own background script after its files were written.
• A few labels crossed by pipes settle only by inference from neighbouring numbers, or at the desk.