Engineering design
A design has to work once it's built, not just look right on paper. Doing that well means working from the real numbers, staying inside every limit, and proving a design before claiming what it does.
Three things an agent has to get right
Work from the real numbers
Loads, sizes and heights have to come from the records, plans and photos; a design built on a guess earns less when tested.
Stay inside every limit
Clearances, ratings, budgets, glare, safe loads and deadlines count; in the solar designs, breaking a hard rule costs half the reward or all of it.
Prove it before claiming it
Most designs run in a simulation or physics model, and only figures and steps a design really delivers may be stated.
They guess the numbers, break a limit, or claim what they haven't shown
Wrong numbers
Tea factory rooftop solar: 67% of attempts assumed the expanded factory would draw more power than its own records support and sized the solar to it; those designs earned 66% to 77% of the best design's ten‑year value.
Row‑house rooftop solar: in 33% of attempts, the agent measured a neighbour's added floor from the overhead images, then kept the older, lower height in its model; one of those designs fell short of a pass.
Limits broken
DIY smart‑lighting plan: 63% of attempts failed the check that no beam reaches the eyes of someone reading in bed, one only because its parts table couldn't be read.
Night repair of a leaking condo floor: in 57% of attempts, the repair wasn't finished by the 01:30 deadline, after time lost drying a base that would never dry, or laying planks twice.
Not proven
Tea factory rooftop solar: 83% of attempts stated a ten‑year value at least 25% above what the grader computes for the same design.
800 kg air handler move: in every attempt, the agent never proved a way to get the unit up to the plant room floor; most handed in with over half their time left, though they could reset and try again at no cost.
Partial credit for nearly every attempt, and a single full solve
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Tea factory rooftop solar and battery design with its business caseEnergy engineering | Real numbers. Solar and a battery are sized for next season's load, worked out from the factory's records and memos, not the recorded one. | 18 | 6 | 92% |
| 800 kg air handler move proved by trial in a 3D physics modelFacilities | Proved by test. By trial in a physics model, the agent proves how two workers can get an 800 kg unit up a ramp unharmed. | 18 | 7 | 44% |
| DIY smart‑lighting plan from notes, photos and one shop's catalogueHousehold | Within limits. Six lights must suit the homeowner's wiring, clear the ceiling fan and keep their beams out of a reader's eyes. | 26 | 8 | 94% |
| Night repair of a leaking condo floor on an in‑world clockHousehold | Within limits. The rebuilt floor must stay flat and flush for six weeks, and the work must be done by 01:30. | 13 | 7 | 100% |
| Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solar | Real numbers. The right house's measurements come from plans, images and photos, and the quote may state only what the design earns. | 15 | 6 | 99% |
| Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineering | Within limits. The fastest legal car can't be bought, so the design has to fit the budget, race day and what suppliers can deliver. | 4 | 5 | 2.0% slower |
Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials