AsapiYour AI engineering assistant
We built the drawings and records of an industrial plant, the kind that show every pipe, valve and instrument. In one job, the AI is a consultant hired at a fixed price to turn faded, hand‑marked scans of the drawings into lists, and each question to the plant's staff costs money. In the other, it works through six months of paperwork from nine teams and draws the changes really made onto the master drawings.
2 jobs · 6 AIs tested. Sonnet 5.5 did best on average, then Opus 5.5 and GPT‑6.1 Sol.
The two jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
Digitise a plant's scanned drawings (5 AIs tested)
Hired at a fixed price, the AI turns an industrial plant's faded, hand‑marked scanned drawings into lists of every instrument, pipe and piece of equipment, and each question it asks the plant's staff comes out of that price.
What the best AI did
- Read every instrument, pipe and piece of equipment off the scans correctly, and stayed within the fixed price.
How each AI scored
Bring a plant's drawings up to date (6 AIs tested)
Six months of paperwork from nine teams says what changed on an industrial plant, and the AI checks each claim against what was really done and draws the true changes onto the plant's master drawings.
What the best AI did
- Judged all 44 possible changes and drew all nine sheets right, where every other AI slipped on both.
How each AI scored
Who did best
Anthropic's Sonnet 5.5, a mid‑sized AI, read every detail on the scans right, and Fable 5.1, tested only on the update, was the only AI to judge every change and draw every sheet right. Opus 5.5 got one detail wrong on the scans and slipped twice on the update. GPT‑6.1 Sol scored 99 on the scans too, but one misread digit got its delivery rejected. GPT‑6 Luna, a small AI, found fewer than half the items.
Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on both jobs.
What the two jobs showed
What they do well
They can read faded, hand‑marked drawings almost without a slip.
Four of the five AIs found all or all but one of the 737 items on the scans, and got at most 18 of their 3,078 details wrong. Sonnet 5.5 read every one right.
They ask the questions that matter, and stay within budget.
Updating the drawings, all six AIs asked the two questions only the plant's staff could answer, kept their questions within the $60 allowed, and left the rest of every sheet as it was.
Where they fall short
One misread digit can sink a delivery.
GPT‑6.1 Sol scored 99 out of 100 on the scans, but misread one digit in a pipe's number and listed a pipe that doesn't exist, an error the plant rejects outright. It never asked the plant about that pipe.
A small AI falls far behind.
GPT‑6 Luna stopped with fewer than half the items on the scans listed, and drew only three of the nine sheets right.
How these jobs were set up and scored
- The plant
- One made‑up unit of an industrial plant, drawn on nine sheets. Each AI worked alone, with the plant's records, a desk that answers questions for a fee and no internet.
- Scores
- The scans job has more than 3,800 checks: every item found, every detail read, the lists matched against the plant's control system and the charges within the price. The update has 65: each of 44 possible changes judged, each of nine sheets drawn and otherwise left alone, and the questions asked within the budget. Scripts score both.
- Passing
- On the scans job, the plant accepts a delivery only if almost nothing in any list is missing or wrong, the lists match the control system exactly, the charges stay within the price, and it has none of the errors the plant rejects outright, such as a pipe that doesn't exist. Three of the five AIs passed, two of them with a score of 99.
- Time and cost
- Each AI had one try at each job it was tested on. The scans took 4 to 35 minutes and the update 10 to 21, and a try cost from 3 cents to $22.05 at published prices.
Every AI's score on every job
| AI | Digitise a plant's scanned drawings | Bring a plant's drawings up to date |
|---|---|---|
| GPT‑6 AstraOpenAI | 99 | 92 |
| GPT‑6 LunaOpenAI | 45 | 83 |
| GPT‑6.1 SolOpenAI | 99 | 97 |
| Opus 5.5Anthropic | 99 | 97 |
| Sonnet 5.5Anthropic | 100 | 97 |
| Fable 5.1Anthropic | – | 100 |
These are the scores out of 100 for the request as a person would write it, one try per AI. Darker means higher. A dash means we didn't test that AI on that job.