Night repair of a leaking condo floor on an in‑world clock

The brief gives the goal and the night; the source, the wet and the depths show only to tests

Role

The owner of a short‑stay condo, alone in the flat the night before a guest checks in, acting only through one command, wa, in a text model of the flat.

Request

Find and stop the leak and rebuild the hall floor so it stays sound for some six weeks of use, with only what is in the unit; tell the buyer what was found and done, leave the guest a note, and be gone before a fixed hour.

Deliverable

The final state of the floor and the flat, and the messages to the buyer and the guest. Leaving the flat ends the attempt.

What the reward depends on

The real source sealed; all wet wood out and the base prepared; a barrier, a supported bottom layer and a floor flat and flush now and after six weeks, its edge sealed; done in time with the rooms as before; complete, honest messages.

Where the requirements are

In the world's replies to the agent's own looks, feels, measurements and tests, and in a published table of how long each action takes. There are no documents, and the buyer and the guest don't reply.

A text model of the flat on an in‑world clock, with no one to ask

Interface

A shell in a fresh Docker container, with the agent's own CLI and one client, wa: look, feel, measure, test the meter, pour plain or coloured water, dry, seal, lift and lay floors, move furniture, message, wait and leave.

Isolation

The world, a state machine with no model calls, runs in a second container, so its code and the grader can't be read. The container reaches only the model route and the world; web search and fetch were off.

The flat

A hall floor of click laminate on hardboard over a sand‑cement screed, soaked in front of the bathroom door; the same laminate in the bedroom and the living room. The shops are shut.

In the unit

Click planks, part of a bag of sand‑cement, one tube of silicone, hand tools and levels, towels, drying appliances, plastic sheet and food colouring; the world counts down every material used.

Hidden state

The leak's source, how far the wet goes under the floor and the depths of the opened hole show only through the agent's own tests and measurements.

Heights

The world tracks the hole's depth and every layer per 50 mm band across the hall: as laid, underfoot now, and after six weeks of use.

People

None to talk to: the agent can message the buyer and leave the guest a note, and neither replies.

Clock

World time moves only when the agent acts or waits, by the minutes in a published table; thinking costs none. The night holds four and a half hours of world time.

Limits

One session of up to 120 minutes, 150 turns and $20 at list prices; no attempt reached a limit.

13 checks of equal weight: 10 by script on the final state, 3 by two judges on the messages

Structure

13 checks in five families: stopping the leak (1), opening and preparing the floor (3), a flat floor that lasts (4), time and the rooms (2), and the messages (3).

World checks

Scripts score 10 checks on the final state: the source sealed, no wet wood left, the base prepared, a barrier, a supported bottom layer, flat and flush now and after six weeks, a sealed edge, done in time, and the rooms as before.

Message checks

Two model judges, Opus 5.5 and GPT‑6.1 Sol, with no tools, read the messages against a fact sheet from the world log that holds the hidden truth. Both are among the models tested.

Rulings

A check passes only when both judges pass it. Splits are ruled by the environment's author: 11 of the 63 message verdicts with only the request.

No free passes

The support, edge, time and rooms checks need a new floor over the whole opening, so an empty visit scores 0; an attempt that sends no message fails all three message checks.

Formula

Reward = checks passed / 13, each worth the same, with no gates. A model's score in a version is the mean of its attempts.

The floor rarely lasted six weeks without the method; given every command, all 20 scored attempts met all 13 checks

With only the request, the median attempt scored 54%, and one of 21, GPT‑6 Astra's first, met all 13 checks; its other two scored 77% and 54%. With a hint the mean rose from 58% to 68%; told the rules, all 20 scored attempts met all 13.

Each dot is one attempt; the line marks the median.

Each model made three attempts in each version, and the table shows their mean. Fable 5.1's safety filter refused all three with a hint and one told the rules, on the first turn; those aren't scored.

With a hint: the request plus a note from a friend who has done flooring jobs, with ten pieces of general advice on the source, the wet, the materials, the packing and the messages.

Told the rules: the request plus every command of one passing method, in order, in about 530 words, with the hidden facts the checks turn on.

With only the request, a model's three attempts differed by up to 46 points.

What the models got wrong

Counted over the 21 attempts with only the request, 7 models × 3, unless a line says otherwise.

1. A floor that doesn't last. 19 of 21 left a new floor that isn't flat and flush after six weeks, and 17 one that isn't flush even on the night; most packed the deeper part of the hole with material that loses height, or left it low.

2. The edge left open. 17 of 21 didn't seal the joint between the new floor and the old all the way along. With the request or a hint, the 9 attempts that sealed a joint that doesn't leak never sealed the whole edge.

3. Out of time. 12 of 21 weren't finished and gone by the night's end, after minutes spent drying a base the world says won't dry, measuring again or laying planks twice.

4. Too much floor opened. GPT‑6 Astra and GPT‑6.1 Sol opened the floor past the edge the world reports dry in 11 of their 12 attempts with the request or a hint, then ran short of plastic or planks.

5. Rooms not put back. 10 of 21 left the bedroom not as it was: skirting off, the bed out, or fewer rows re‑laid than lifted.

6. Messages beyond what was seen. 9 of 21 sent a message that states something false or hides a known problem, such as an edge called sealed after the silicone ran out, or a floor called replaced before work began.

What an attempt costs

Length

With only the request, the median attempt took 98 seconds of real time and 13 tool calls (10 to 26 across the 21).

Cost

Claude Code's own figures at list prices for its 12 attempts with only the request: $0.06 to $1.45. Agent model only: the judges aren't included, and the world makes no model calls.

OpenAI's estimates

The runner's estimates for the 9 Codex CLI attempts, their logged tokens at list prices: $0.07 to $0.75. All attempts ran on plan sign‑ins, so none was billed per call.

What you license

With a licence

Revision 1 in all three versions: the world with its state machine and the wa client, the grader with the judges' instructions and fact‑sheet builder, the self‑test's scripted reference walk, and the grader's report on each of the 59 scored attempts, as baseline results.

Format

The agent runs in a Docker container; the world runs in a second container, reached through one port. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The world, brief and grader aren't published.

What we haven't measured, and what's uneven

• Three attempts per model and version; a difference of a check or two between two models' means is small.
• Fable 5.1's safety filter refused the prompt with a hint in all three attempts, and one told the rules, on the first turn. They weren't made again, so it has no score with a hint.
• A review fixed four issues in the world and the grader. Only Opus 5.5's three attempts with a hint were made again on the fixed world; every attempt was re‑scored on the fixed grader.
• The world's trade facts, such as how fast packing sinks or mortar sets, were set by the task's design and not tested against a real floor.
• The two message judges are among the models tested and judged their own messages; splits were ruled by the environment's author.
• Told the rules gives every command of one passing method, and all 20 scored attempts met all 13 checks: that version checks fairness, not difficulty.
• The self‑test plays the reference walk, which passes all 10 world checks, and each known shortcut, which fails its check; an empty visit scores 0.