AsapiThe AI Agent Trials
Asapi / Catalogue / Photographed P&ID sketches digitised into instrument, line and equipment lists
Environment datasheet · Engineering records

Photographed P&ID sketches digitised into instrument, line and equipment lists

ID tb‑tasks/pid‑digitise‑sketch · revision 1 · results as of 10 November 2026

As a digitisation consultant on a fixed price, the agent reads a plant unit's pencil sketches, seven sheets split over two pages, into an index of about 400 instruments, a list of about 230 lines and about 100 equipment items, and checks the control system's out‑of‑date export against them.

Checks
3,836, 12 gates; each invented row adds a failed check
Tested
5 models, one attempt each
Best
99% (GPT‑6 Astra)
Median
99%
Full solves
0 of 5
Median attempt
18 min
What it tests

About 730 plant items read off pencil sketches, at a fixed price

Multimodal records

Every row is read from 17 photographed pencil pages in the drafter's own symbols, with look‑alike digits to tell apart.

Clarifying with people

A scripted query desk answers only what is asked, and the client's practice on control‑system points is learnable only from it.

When to act or escalate

Desk checks are charged against the fixed price, so checking everything costs more than the job pays, and a guessed digit fails.

Catching errors

The control system's out‑of‑date export writes some tags another way; only the eight real discrepancies may be reported.

Unstated requirements

The client's practice on which instruments get a control‑system point is found only through the priced query desk.

How it works

17 photographed pages, a priced query desk and no OCR

The agent's world

A shell in a Linux container with Python and image libraries but no OCR; pages are read by cropping and enlarging regions.

The drawings

A legend and nine sheets, seven split over two pages at a match line, with smudged tags, site clutter and labels crossed by pipes.

Limits

One session of 90 minutes, against an expert's estimate of six hours.

The reward

One point per row and field across the three lists, eight reconciliation rows and a 10‑point price check; each invented row adds a failed check.

Results

Four attempts scored 99%, and one stopped early at 39%

Each dot is one attempt, and the line marks the median.

MedianFully solved
5 modelsOne attempt each99%0 of 5
ModelReward
GPT‑6 AstraOpenAI99%
Opus 5.5Anthropic99%
Sonnet 5.5Anthropic99%
GPT‑6.1 SolOpenAI99%
GPT‑6 LunaOpenAI39%

Three of the four also passed the task's own bar; the fourth delivered three rows that don't exist.

What models get wrong

They misplace pump‑group connections, misread labels unchecked, and trust the export

Multimodal records

  1. Every attempt that read the drawings put a gauge's connection or a line's end on the wrong branch of a pump group, on rows never checked at the desk.

When to act or escalate

  1. One misread three labels and delivered three rows that don't exist, errors the engagement letter rejects outright, without checking any at the desk.

Catching errors

  1. One built the index from the out‑of‑date export, missing 168 instruments, 29 of them safety instruments, and stopped after 10.5 minutes.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.05 to $29.04 for the agent model; grading is a script, with no model calls.

What's in it

The photographed pages and legend, the engagement letter and export, the query desk, the grader and answer key, and a passing reference solution.

Format

Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the desk as a root‑owned script.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The plant, desk, export, price and grader match the scanned‑drawings environment, so the two differ only in the drawings. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.