AsapiThe AI Agent Trials
Asapi / Catalogue / Scanned P&ID set digitised into instrument, line and equipment lists
Environment datasheet · Engineering records

Scanned P&ID set digitised into instrument, line and equipment lists

ID tb‑tasks/pid‑digitise · version 4.0.1 · tested 31 October 2026 on 4.0.0, regraded on 4.0.1

As a digitisation consultant on a fixed price, the agent reads a plant unit's ten scanned sheets into an index of about 400 instruments, a list of about 230 lines and about 100 equipment items, and checks the control system's out‑of‑date export against them.

Checks
3,836, 12 gates; each invented row adds a failed check
Tested
5 models, one attempt each
Best
100% (Sonnet 5.5)
Median
99%
Full solves
1 of 5
Median attempt
17 min
What it tests

About 730 plant items read off scanned drawings, at a fixed price

Multimodal records

Every row is read from ten scanned sheets, some faded or hand‑marked, where no tag is readable at whole‑sheet size.

Clarifying with people

A scripted query desk answers only what is asked, and the client's practice on control‑system points is learnable only from it.

When to act or escalate

Desk checks are charged against the fixed price, so checking everything costs more than the job pays, and a guessed digit fails.

Catching errors

The control system's out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported.

Unstated requirements

The client's practice on which instruments get a control‑system point is found only through the priced query desk.

How it works

Ten scanned sheets, a priced query desk and no OCR

The agent's world

A shell in a Linux container with Python and image libraries but no OCR; scans are read by cropping and enlarging regions.

The scans

A legend and nine drawings of 9,933 by 7,016 pixels, for two process trains, cooling water and a flare drum, some faded or hand‑marked.

Limits

One session of 90 minutes, against an expert's estimate of six hours.

The reward

One point per row and field across the three lists, eight reconciliation rows and a 10‑point price check; each invented row adds a failed check.

Results

Four attempts scored 99% or more, and one stopped early at 45%

Each dot is one attempt, and the line marks the median.

MedianFully solved
5 modelsOne attempt each99%1 of 5
ModelReward
Sonnet 5.5Anthropic100%
Opus 5.5Anthropic99%
GPT‑6.1 SolOpenAI99%
GPT‑6 AstraOpenAI99%
GPT‑6 LunaOpenAI45%

Three of the four also passed the task's own bar; the fourth delivered a line that doesn't exist.

What models get wrong

They connect gauges to the nearest pipe, misread a digit unchecked, and trust the export

Multimodal records

  1. 60% put a field gauge's connection on the nearest pipe or the wrong line, on rows no desk check covered.

  2. Two numbered a dense pump group's branch lines in the wrong order, putting line ends and gauges on the wrong lines.

When to act or escalate

  1. One misread a digit in a line number and delivered a line that doesn't exist, an error the engagement letter rejects outright, without checking that service at the desk.

Catching errors

  1. One built the index from the out‑of‑date export, missing 161 instruments, among them all 23 emergency‑shutdown level switches, and stopped after 4 minutes.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.03 to $22.05 for the agent model; grading is a script, with no model calls.

What's in it

The scans and legend, the engagement letter and export, the query desk, the grader and answer key, and a passing reference solution.

Format

A Harbor task in a Linux container, with the desk as a root‑owned script.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The sketched‑drawings environment uses the same plant, desk, export, price and grader, with pencil sketches instead of scans. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.