P&ID as‑built backlog from nine teams' records, drafted in DXF

Each backlog item names its source document, not whether the change happened

Role

A contract drafter for a plant whose piping and instrumentation drawings, held as a nine‑sheet CAD master, are six months behind.

Request

Weigh each backlog item against the plant's records, decide update, no change or hold with a reason, and draw each real change onto the master.

Deliverable

The nine CAD sheets as delivered, changed or not, and a file with one decision, reason and sheet list for each backlog item and each change found only in the records.

What the reward depends on

All 44 decisions right; each sheet's connections equal to the original plus exactly its expected changes, with nothing else moved; the two questions no record answers asked within budget.

Where the requirements are

In about 700 entries from nine teams, in their own formats, which don't always agree, and in the engagement letter's rules for weighing records.

Nine CAD sheets, nine teams' records and a question budget

Interface

A shell in a Linux container with Python and a DXF library, a plotter for sheets and a priced query desk; no CAD application. Codex ran with web search off; Claude Code's web tools were on for Fable 5.1, Opus 5.5 and Sonnet 5.5, and none used them.

The master

Nine DXF sheets whose pipes, valves and instruments connect by geometry alone; the drawing stores no connections.

Records

The backlog and about 700 entries: work orders, stores issues, permits, change requests, inspections, a contractor's list of finished work, the shift log, fitters' marked‑up prints and emails.

Query desk

Fixed code answers six kinds of question per backlog item from a table; each answer is charged against a fixed budget.

Limits

One session of 90 agent minutes, against an expert's estimate of five hours.

65 checks, with each sheet's connections rebuilt from its geometry; one miss fails the delivery

Structure

44 decision checks (decision, reason and sheets), 9 drafting checks, one per sheet, on its rebuilt topology, 9 that nothing else on a sheet moved or changed, 2 on the questions no record answers and 1 on the budget.

Who decides

The grader compares the decisions with the key, rebuilds each delivered sheet's connections from its geometry and compares them with the original plus that sheet's expected changes; script only.

The task's pass

The task's own pass, kept beside the reward, needs all 65 checks. A valve off its pipe or at a pipe junction fails, and existing items may move only within a small tolerance.

Formula

Reward = checks passed / 65, from 0 to 1; the task's own pass needs all 65. Writing nothing scores under 2%, the budget check only.

One model passed; three missed only a drain the contractor's list claimed was drawn

Fable 5.1 met all 65 checks. GPT‑6.1 Sol, Opus 5.5 and Sonnet 5.5 each scored 97% and failed on the same drain; GPT‑6 Astra scored 92% and GPT‑6 Luna 83%.

Each dot is one attempt; the line marks the median.

One attempt per model, six models.

Every model asked the two questions no record answers and stayed within the budget.

What the models got wrong

Counted over the 6 models, one attempt each.

1. A drain taken on trust. 5 of 6 accepted a claim on a contractor's list of finished work that a drain was already drawn, without finding the pipe it describes; for three of them it was the only miss.

2. A gauge held for no reason. 2 of 6 held an instrument job for a site check because two records give its tag two ways, though the client's rules make the instrument index, which lists one of the two tags, decide.

3. Drawn on the wrong valve. 1 of 6 called a change request already drawn from the note that closed it, without opening the sheet, and drew a like‑for‑like valve swap as a change, on a different valve from the one replaced.

4. Valves off the pipe. 1 of 6 placed new valves at rough positions or at pipe junctions, so they didn't connect; four sheets failed on placement alone, and with its wrong decisions it failed 6 of 9.

What an attempt costs

Length

The median attempt took 11 minutes and about 54 tool calls (22 to 79 across the 6).

Cost

The median attempt cost $3.05, from $0.07 to $11.95 across the 6. All six are estimates from token counts, since every attempt ran on a plan sign‑in not billed per call; GPT‑6 Luna's also uses a derived price, as it has none published.

What you license

With a licence

Version 1.0.0: the CAD master and plots, the backlog and records, the engagement letter and drafting standard, the query desk, the grader and key, a reference solution that passes, and the grader's report on each of the 6 attempts, as baseline results.

Format

A Harbor task in a Linux container, with the desk as a root‑owned script.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The sheets, key and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model.
• The task's own pass needs all 65 checks, so a single missed drain fails an otherwise correct delivery.
• The desk answers from a fixed table, the same for every agent.