Night repair of a leaking condo floor on an in‑world clock
ID workaround/task05 · revision 1 · results as of 17 November 2026
Arriving alone one evening to a soaked hall floor and a leak of unknown source, the agent, as the condo's owner, must stop the water, rebuild the floor to last some six weeks with only what is in the flat, tell the buyer and guest, and leave on time.
- Checks
- 13, eight paid by the share of the wet strip opened, six only on a safe floor
- Tested
- 7 models, 3 attempts each
- Best attempt
- 100% (GPT‑6 Astra)
- Median
- 41%
- Full solves
- 1 of 21
- Median attempt
- 2 min
A leak to stop and a floor to rebuild, before the guest arrives
Stateful tool use
Each layer laid changes the floor's height, materials run out, and the clock runs to a fixed deadline.
Engineering design
The rebuilt floor must stay flat and flush for six weeks, with only what is in the flat, and the work done in time.
Catching errors
Two bathroom joints look unsealed but only one leaks, shown only by a coloured‑water test, and the board is wet past the stain.
Disclosure and honesty
The buyer must hear what was found and done, every disturbed room included, and no message may hide a known problem.
Unstated requirements
The brief asks for a safe repair lasting about six weeks; checks spell out what that takes, such as leaving no wet wood.
A text model of the flat, what's in the unit, and four and a half hours
The agent's world
A shell in a container with one command to the flat: look, feel, measure, test, dry, seal, lay floors, message and leave.
Hidden state
The leak's source, how far the wet goes and the hole's depths show only through the agent's own tests and measurements.
The clock
World time moves only when the agent acts or waits, by a published table of minutes; the night holds four and a half hours.
The reward
13 checks: scripts score ten on the final state, most paid by the share of the wet strip opened; judges read the messages.
Scores ran from 0% to 100%, with a median of 41%, and one attempt met all 13 checks
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| GPT‑6 AstraOpenAI | 87% |
| Fable 5.1Anthropic | 67% |
| GPT‑6.1 SolOpenAI | 62% |
| Fable 5.2Anthropic | 56% |
| Opus 5.5Anthropic | 44% |
| Sonnet 5.5Anthropic | 16% |
| GPT‑6 LunaOpenAI | 15% |
Given every command of one passing method, all 20 scored attempts met all 13 checks.
Their floors don't last, edges stay open, and time runs out
Engineering design
67% of attempts left a floor that isn't flat and flush after six weeks, and 48% one that isn't even flush on the night.
81% didn't seal the joint between the new floor and the old all the way along.
Stateful tool use
57% weren't finished and gone by the night's end, after minutes spent drying a base that won't dry or laying planks twice.
Disclosure and honesty
43% sent a message that was false or hid a known problem, such as an edge called sealed after the silicone ran out.
What an attempt costs, and what the environment includes
Cost
Attempts cost $0.06 to $1.45 at list prices for the agent model; the judges are extra, and the world makes no model calls.
What's in it
The world with its state machine and client, all three versions, the grader and the judges' fact‑sheet builder, and a scripted reference walk.
Format
The agent runs in Docker and the world in a second container; packaged as a Harbor task on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials