800 kg air handler move proved by trial in a 3D physics model

The brief gives the goal and three conditions; the method has to be found by trial in the model

Role

The rigging engineer for a hospital plant room fit‑out, acting only through one command, wa, in a 3D model of the site with the same sizes, weights, floors and forces.

Request

Prove in the model how the crew moves the unit from the store room, along the corridor and up into the plant room, seated on its plinth and facing as its drawing shows, with no mark on it or the floors; then write the method statement they'll follow.

Deliverable

The proved move, read from the world's event log and final state, and a plain‑text method statement handed in through the same command, which ends the attempt.

What the reward depends on

Each stage of the route reached, from the first lift to the unit seated on its pins, and reached with no harm; the unit turned end for end; a clear walkway; a statement whose final state matches the model and that names what carried the unit.

Where the requirements are

In the world: the unit's plate, the workers' cards, the plinth drawing, the tape and the reply to each command. Pushing the unit up the ramp fails, and a way that works has to be found by trial.

A 3D physics model of the site, its kit and two code‑played workers

Interface

A shell in a fresh Docker container, with the agent's own CLI and one client, wa, that sends each command to the 3D world in a headless browser outside it: walk, look, measure, pick and place kit, direct the workers, wait, reset the model and report.

Isolation

The container reaches only the model route and the world's control port, so the world code and the grader can't be read. Web search and fetch were off.

The site

A store room, a corridor that holds the kit, the plant room door, a landing and a short ramp up to the plant room floor, with the plinth at its far end. Sizes, weights, floors and forces follow a physics model.

The unit

New, 800 kg and 1.8 m long, starting the wrong way round for its plinth. Any scratch, dent or mark on it or the floors is harm.

Kit

Skates, jacks, a pry bar, blocks and timber packers, aluminium boards, pipes, floor sheets, blankets, a rope, a trolley and a pallet truck, beside machines that can't be used here. Plates and cards give ratings and limits.

Workers

Two, played by code: they carry out commands given in one fixed word order, within the safe limits on their cards, and the same command in the same state always gets the same reply. Nobody else answers.

Replies

Each command is answered in words; pictures of the agent's view come back on request. With only the request, the median attempt opened 18.

Clock

The world has its own clock, moved by actions and waits, which no check reads. The brief's two hours are real time.

Limits

One session of up to 120 minutes and $20 at list prices, with no turn cap; one attempt with a hint reached the $20 cap.

Variants

Three more variants of the same world and checks change what the agent gets back: words with no pictures; words plus the site's physics code to read; or pictures alone, a red tint marking a refused command.

18 checks of equal weight, all scored by script from the world's log and final state

Structure

Seven stages, from raised off the store room floor to seated on its pins, each with two checks: reached, and reached with no harm on the way. Four more: the unit faces the right way, a clear walkway, and two on the statement.

Families

Five: out of the store room (4), to the door and the turn (4), up the ramp and onto the plinth (6), the finished site (2) and the method statement (2).

Harm

Any mark on the unit or the floors, a lifted tile, a drop, a panel strike, too much tilt, a roller dragged out under load, an overloaded worker, an end left on a bar alone, or a runaway on the ramp.

Best trial

Each reset starts a fresh trial, and the trial with the most checks passed counts. The finished‑site and statement checks need the unit seated in the last trial.

Statement

Scripts read two things: its claim about the unit's final state, which must match the model and can't call the move unmarked after harm, and a word for each carrier that moved the unit.

Formula

Reward = checks passed in the best trial / 18, each worth the same, with no gates and no model judges. An attempt that hands in no statement fails the two statement checks.

No attempt got the unit up the ramp without the method; given it step by step, four of seven met all 18 checks

With only the request, no attempt got the unit to the plant room floor; the best two, at 44%, reached the plant room door with the unit turned. With a hint the mean went from 29% to 31%; told the rules, to 68%, and four met all 18 checks.

Each dot is one attempt; the line marks the median.

Each model made one attempt in each version. GPT‑6.1 Sol's attempt told the rules is its second, made after the method's turn angles were rounded to one decimal place.

With a hint: the request plus a rigging supervisor's note with eight pieces of advice and five ways round an obvious method that fails.

Told the rules: the request plus the supervisor's method in 22 steps, with every command written out, the first words of each expected reply and the statement to hand in.

In the other three variants, with only the request, the best attempts scored 33% with words only, 94% with the code to read and 22% with pictures alone; told the rules, 3, 5 and 0 of 7 met all 18 checks.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request, unless a line says otherwise.

1. No way up the ramp. 7 of 7 attempts never brought the unit up to the plant room floor, so each lost at least 10 of the 18 checks; with a hint, none did either.

2. Stopping with time left. 6 of 7 handed in before an hour of their two had passed, all by choice. Three judged the way over the ramp unsafe or uncertain and wrote a hold point into the statement, though a reset makes a failed trial free.

3. No turn. 5 of 7 never turned the unit end for end, so it couldn't have faced the right way on the plinth; two concluded that no turn was possible with the kit on site.

4. Never out of the store room. 2 of 7 didn't get the unit out: GPT‑6 Luna surveyed for two minutes and handed in, and Sonnet 5 raised the unit, then in 58 minutes never moved it out, writing unproven steps into its statement as the method.

5. Claims beyond the model. Told the rules, Sonnet 5 lost its place in the method and GPT‑6 Luna stopped after one lift; both handed in the method's prepared statement that the unit was seated, while it stood at the door or in the store room.

What an attempt costs

Length

With only the request, the median attempt took 29 minutes of real time and 165 tool calls (24 to 431 across the 7).

Cost

Claude Code's own figures at list prices for its 4 attempts with only the request: $1.96 to $17.09. Agent model only: the world and the grader make no model calls.

OpenAI's estimates

The runner's estimates for the 3 Codex CLI attempts, their logged tokens at list prices: $0.24 to $18.67. All attempts ran on plan sign‑ins, so none was billed per call.

What you license

With a licence

Revision 1 in all three versions and four variants: the 3D world with its physics and the wa client, the read‑only site code, the grader, the self‑test's scripted reference walk, and the grader's report on each of the 84 attempts, as baseline results.

Format

The agent runs in a Docker container; the 3D world runs in a headless browser outside it, reached through one port. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The world, brief and grader aren't published.

What we haven't measured, and what's uneven

• Each model made one attempt in each version, so a model's score includes the variation between attempts.
• The agent's own character can place kit, a third pair of hands beside the two workers, though the brief allows no extra crew. No check tests this, and the method told the rules uses it.
• The two statement checks read words: the claimed final state and a word for each carrier. Both count only once the unit is seated in the last trial.
• The method told the rules first gave angles to six decimal places; one attempt typed an angle a millionth of a degree off, and its later steps landed elsewhere, a fault of the world rather than the model. That attempt wasn't made again.
• With pictures alone, small lifts and drops don't show, and agents repeated commands that dropped the unit, one 103 times; no attempt in that variant turned the unit or reached the plinth.
• The self‑test plays 51 scenarios through the engine, valid plans and wrong ones, with no model calls; each behaves as designed.