AsapiThe AI Agent Trials
Asapi / Capabilities / Engineering design
Capability · 6 environments

Engineering design

A design has to work once it's built, not just look right on paper. Doing that well means working from the real numbers, staying inside every limit, and proving a design before claiming what it does.

What it takes

Three things an agent has to get right

Work from the real numbers

Loads, sizes and heights have to come from the records, plans and photos; a design built on a guess earns less when tested.

Stay inside every limit

Clearances, ratings, budgets, glare, safe loads and deadlines count; in the solar designs, breaking a hard rule costs half the reward or all of it.

Prove it before claiming it

Most designs run in a simulation or physics model, and only figures and steps a design really delivers may be stated.

What models get wrong

They guess the numbers, break a limit, or claim what they haven't shown

Wrong numbers

  1. Tea factory rooftop solar: 67% of attempts assumed the expanded factory would draw more power than its own records support and sized the solar to it; those designs earned 66% to 77% of the best design's ten‑year value.

  2. Row‑house rooftop solar: in 33% of attempts, the agent measured a neighbour's added floor from the overhead images, then kept the older, lower height in its model; one of those designs fell short of a pass.

Limits broken

  1. DIY smart‑lighting plan: 63% of attempts failed the check that no beam reaches the eyes of someone reading in bed, one only because its parts table couldn't be read.

  2. Night repair of a leaking condo floor: in 57% of attempts, the repair wasn't finished by the 01:30 deadline, after time lost drying a base that would never dry, or laying planks twice.

Not proven

  1. Tea factory rooftop solar: 83% of attempts stated a ten‑year value at least 25% above what the grader computes for the same design.

  2. 800 kg air handler move: in every attempt, the agent never proved a way to get the unit up to the plant room floor; most handed in with over half their time left, though they could reset and try again at no cost.

Results

Partial credit for nearly every attempt, and a single full solve

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

Hover a dot for its model and reward.

The 6 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Tea factory rooftop solar and battery design with its business caseEnergy engineeringReal numbers. Solar and a battery are sized for next season's load, worked out from the factory's records and memos, not the recorded one.18692%
800 kg air handler move proved by trial in a 3D physics modelFacilitiesProved by test. By trial in a physics model, the agent proves how two workers can get an 800 kg unit up a ramp unharmed.18744%
DIY smart‑lighting plan from notes, photos and one shop's catalogueHouseholdWithin limits. Six lights must suit the homeowner's wiring, clear the ceiling fan and keep their beams out of a reader's eyes.26894%
Night repair of a leaking condo floor on an in‑world clockHouseholdWithin limits. The rebuilt floor must stay flat and flush for six weeks, and the work must be done by 01:30.137100%
Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solarReal numbers. The right house's measurements come from plans, images and photos, and the quote may state only what the design earns.15699%
Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineeringWithin limits. The fastest legal car can't be bought, so the design has to fit the budget, race day and what suppliers can deliver.452.0% slower

Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.