AsapiThe AI Agent Trials
Asapi / Catalogue / P&ID as‑built backlog from nine teams' records, drafted in DXF
Environment datasheet · Engineering records

P&ID as‑built backlog from nine teams' records, drafted in DXF

ID tb‑tasks/pid‑asbuilt · version 1.0.0 · tested 2 November 2026

As a contract drafter, the agent decides for each of 42 backlog items, and for changes found only in the records, whether the plant's drawings need an update, then draws each real change onto the CAD master, asking the client's engineer only what no record settles.

Checks
65, 1 point each; the task's own pass needs all
Tested
6 models, one attempt each
Best
100% (Fable 5.1)
Median
97%
Full solves
1 of 6
Median attempt
11 min, 54 tool calls
What it tests

A six‑month backlog of drawing updates, weighed against nine teams' records

Catching errors

44 decisions turn on records that disagree, describe cancelled or temporary work, repeat each other, or claim a change is already drawn.

Multimodal records

Changes are traced through about 700 records in nine teams' formats and drawn onto nine CAD sheets that store no connections.

Clarifying with people

A scripted engineer answers questions charged against a budget, and two items fail unless the one decisive question was asked.

When to act or escalate

Holding an item the records settle is wrong, and so is drawing a guess; one change can't be placed at all and must be held.

How it works

Nine CAD sheets, about 700 records and a priced query desk

The agent's world

A shell in a Linux container with Python, a DXF library, a plotter for sheets and a priced query desk; no CAD application.

The records

The backlog and about 700 entries: work orders, permits, inspections, a contractor's list of finished work, the shift log, marked‑up prints and emails.

Limits

One session of 90 agent minutes, against an expert's estimate of five hours; each answer from the desk is charged against a fixed budget.

The reward

65 checks by script: 44 decisions, each sheet's connections rebuilt from its geometry, nothing else moved, two questions asked, and the budget.

Results

Every attempt scored 83% or more, and one met all 65 checks

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each97%1 of 6
ModelReward
Fable 5.1Anthropic100%
GPT‑6.1 SolOpenAI97%
Opus 5.5Anthropic97%
Sonnet 5.5Anthropic97%
GPT‑6 AstraOpenAI92%
GPT‑6 LunaOpenAI83%
What models get wrong

They trust a contractor's list, hold what the rules decide, and draw off the pipe

Catching errors

  1. 83% accepted a contractor's claim that a drain was already drawn, without finding the pipe it describes; for three it was the only miss.

  2. One called a change request already drawn from the note that closed it, and drew a like‑for‑like valve swap as a change, on the wrong valve.

When to act or escalate

  1. Two held an instrument job for a site check over two spellings of its tag, though the client's rules make the instrument index decide.

Multimodal records

  1. One placed new valves roughly or at pipe junctions, so they didn't connect; four sheets failed on placement alone.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.07 to $11.95 for the agent model; grading is a script, with no model calls.

What's in it

The CAD master, the backlog and records, the engagement letter and drafting standard, the query desk, the grader and key, and a reference solution.

Format

A Harbor task in a Linux container, with the query desk as a root‑owned script.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.