P&ID as‑built backlog from nine teams' records, drafted in DXF
ID tb‑tasks/pid‑asbuilt · version 1.0.0 · tested 2 November 2026
As a contract drafter, the agent decides for each of 42 backlog items, and for changes found only in the records, whether the plant's drawings need an update, then draws each real change onto the CAD master, asking the client's engineer only what no record settles.
- Checks
- 65, 1 point each; the task's own pass needs all
- Tested
- 6 models, one attempt each
- Best
- 100% (Fable 5.1)
- Median
- 97%
- Full solves
- 1 of 6
- Median attempt
- 11 min, 54 tool calls
A six‑month backlog of drawing updates, weighed against nine teams' records
Catching errors
44 decisions turn on records that disagree, describe cancelled or temporary work, repeat each other, or claim a change is already drawn.
Multimodal records
Changes are traced through about 700 records in nine teams' formats and drawn onto nine CAD sheets that store no connections.
Clarifying with people
A scripted engineer answers questions charged against a budget, and two items fail unless the one decisive question was asked.
When to act or escalate
Holding an item the records settle is wrong, and so is drawing a guess; one change can't be placed at all and must be held.
Nine CAD sheets, about 700 records and a priced query desk
The agent's world
A shell in a Linux container with Python, a DXF library, a plotter for sheets and a priced query desk; no CAD application.
The records
The backlog and about 700 entries: work orders, permits, inspections, a contractor's list of finished work, the shift log, marked‑up prints and emails.
Limits
One session of 90 agent minutes, against an expert's estimate of five hours; each answer from the desk is charged against a fixed budget.
The reward
65 checks by script: 44 decisions, each sheet's connections rebuilt from its geometry, nothing else moved, two questions asked, and the budget.
Every attempt scored 83% or more, and one met all 65 checks
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.1Anthropic | 100% |
| GPT‑6.1 SolOpenAI | 97% |
| Opus 5.5Anthropic | 97% |
| Sonnet 5.5Anthropic | 97% |
| GPT‑6 AstraOpenAI | 92% |
| GPT‑6 LunaOpenAI | 83% |
They trust a contractor's list, hold what the rules decide, and draw off the pipe
Catching errors
83% accepted a contractor's claim that a drain was already drawn, without finding the pipe it describes; for three it was the only miss.
One called a change request already drawn from the note that closed it, and drew a like‑for‑like valve swap as a change, on the wrong valve.
When to act or escalate
Two held an instrument job for a site check over two spellings of its tag, though the client's rules make the instrument index decide.
Multimodal records
One placed new valves roughly or at pipe junctions, so they didn't connect; four sheets failed on placement alone.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.07 to $11.95 for the agent model; grading is a script, with no model calls.
What's in it
The CAD master, the backlog and records, the engagement letter and drafting standard, the query desk, the grader and key, and a reference solution.
Format
A Harbor task in a Linux container, with the query desk as a root‑owned script.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials