How the environments are built and graded

The words on these pages

Environment

A simulated world (files, services, people, and a clock where time matters), a request, and a grader.

Attempt

One model's go at one environment, from the request to its last action, in one version of the request.

Versions

With only the request: as a user would give it. With a hint: plus a short pointer to what matters.

Told the rules

The request plus the rules or method the job depends on. In 7 of 36 environments it also gives facts from the files or people; in three of those, figures the grader checks.

Variant

A copy of an environment with different data, people and facts under the same design.

Check and reward

A check is one requirement the grader tests. The reward is the weighted share of checks met, from 0 to 1, or a cost model's score where a plan is priced.

Full solve

An attempt that met every requirement. Two attempts scored 1 but missed a requirement that carries no weight in the reward, so they don't count as full solves.

Gates and pass/fail

Some environments zero part of the reward when a make‑or‑break check fails. In 17, the grader also reports its own pass/fail. These pages use the reward; the samples report both.

Capability

Each environment is tagged with the capabilities its reward depends on, checked against its grader; 55 of 56 have two or more.

What's in each world

Data

Synthetic businesses and households with their own files: up to 4,490 at a time in mixed formats (16,554 over one environment's four acts), or case systems with tens of thousands of records. Planted facts, errors and look‑alikes are set by design and recorded in the answer key.

Services

Web apps, booking and supplier sites, banks, registration portals and simulators, each holding state the agent can change; every action is logged.

Simulated people

LLM‑voiced or scripted people who answer only what they're asked; the voice can run on a model you choose. See clarification with simulated users.

Time

Single sessions with a fixed in‑world date; in‑world clocks that move with each action; or simulated days, weeks and months in many sessions, each opened by mail or a schedule, with only the agent's own files carried between them.

Checks before any attempt

Self‑tests check that the answer key passes and the starting state fails; in some environments, gates prove that every deciding record can be found by reasonable searches.

How we tested 23 models

Attempts

583 with only the request, 322 with a hint and 332 told the rules, plus 1 in a fourth version of one environment.

Models

23 models from eight makers, 1 to 15 per environment.

OpenAI

GPT‑5.6 Luna, GPT‑5.6 Sol, GPT‑6 Astra, GPT‑6 Luna, GPT‑6 Sol, GPT‑6.1 Sol.

Anthropic

Fable 5.1, Opus 4.8, Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5.

Google

Gemini 4 Argon.

Open‑weight

Alibaba (Qwen3‑235B, Qwen3.8‑27B); DeepSeek (DeepSeek V3.2, DeepSeek V4.1 Flash); Moonshot (Kimi K2.5, Kimi K3); NVIDIA (Nemotron Super 3); Zhipu (GLM‑5, GLM‑5.3, GLM‑5.3‑Flash).

Harnesses

Each model ran in an agent harness: OpenAI's in Codex CLI, almost all of Anthropic's in Claude Code, Google's in Antigravity CLI, and open‑weight models in Kimi CLI, Qwen Code and other harnesses. Most models used one harness throughout, so model and harness effects are mixed.

Sandboxes

Most attempts ran in a Docker container; in 11 environments, some or all ran locally, in a macOS sandbox or a plain working folder, and the two course registrations use a portal harness.

Budgets

Each attempt had limits set per environment, on wall‑clock time and usually on turns or spend; for example, 150 turns, 120 minutes and $10–20 for the app builds.

Versions

With only the request on all 56; with a hint on 31; told the rules on 36.

Repeats

One attempt per model and version on 46 environments. Repeated attempts on 10: three per model, except at the coffee roaster's materials planning, where models made one or two.

When

September to early November 2026, so the results are for the model versions available then.

Scripts first, judges where needed

Script checks

Pooled over attempts with only the request, code decides 88% to 89% of check results and 83% to 84% of points, or 78% of points averaged over environments. In 11, judges decide most points.

What scripts check

The final state of bookings, payments and registrations; documents field by field, or byte for byte where the format is fixed; figures and dates exactly.

Hidden tests

The two spec‑to‑code environments run hidden test suites (466 and 919 tests) on inputs the agent never sees.

Simulators

Most long‑horizon environments replay the action log through the simulator after the last session and check that the replay reproduces the recorded run.

Cost models

Three environments price the agent's plan: two score it from a baseline plan (0) to the best of tens of thousands of candidates a search found (1); in the third, profit is 70% of the reward.

Tester models

App builds are operated by a tester model in a real browser, limited to page actions, while scripts probe security, restarts, recovery and clock changes.

LLM judges

Where a script can't decide, judges answer fixed yes/no questions: usually two from different model families (for example Opus 4.8 and GPT‑5.6 Luna), or two blind readers with reference and near‑miss verdicts mixed into every batch. Both must agree. Judges can run on models you choose.

No judge

23 of 56 environments are graded by code alone.

Rulings

Judge splits are settled by written ruling: by the environment's author, Claude for the author, the coordinator in the bill reviews and trip rebookings, Opus 5.5 in three app builds, or the grading session in others. A few rulings from 15 October have no recorded author.

Overturned verdicts

Rulings, stored with the grades, also overturned 64 verdicts on which both judges agreed, in 10 environments.

Checks on the reward

Difficulty

With only the request, 37 of 583 attempts (6.3%) were full solves and 96% earned some reward; the middle half of attempts scored 48–87%.

Where full solves happen

34 of 140 attempts on the 20 environments without a told‑the‑rules version, mostly long‑horizon and coding; 3 of 443 on the other 36.

Solvable

In the 31 environments where the same models were tested on both, full solves rose from 3 to 73 of 323 attempts once told the rules, and the median reward from 65% to 91%. In the 25 whose told‑the‑rules version gives no facts: 2 to 48 of 250.

Best when told the rules

The best attempt told the rules reaches 95% or more on 27 of 36 environments, and a full solve on 14. Eight of the nine below 95% are app builds.

Training run

DeepSeek 14B, trained on a simplified version of one environment, went from 20 to 97 out of 100, out of sample. It is the one training run so far.

Right for the right reason

In 5 environments, a correct decision earns full credit only if the deciding fact was in the attempt's log before it was made.

Doing nothing

In 25 environments the grader's own self‑test requires that doing nothing scores 0, and checks that a mistake wasn't made pass only for attempts that did the work. The exceptions are under limitations.

Reference solutions

They fully solve 42 of 56 environments. Of 12 app builds, 10 have no stored reference app and 2 one graded in part; in two long jobs the reference passes but misses a few checks. In 18, the package is gated: the reference must pass and doing nothing must fail.

Shortcuts

We graded degenerate strategies in 55 of 56. Most score little: last month's report copied forward scores 3–5%, returning every listing scores 0. Doing nothing at the coffee roaster's order desk scores about half, from restraint checks, though every act fails.

Answer hunting

In our attempts' logs, one model, DeepSeek V4.1 Flash, searched its container, a simulator's service and the web for answers or the answer key in several attempts. It found nothing useful, and its scores stand.

Leak checks

In 8 environments, canary strings planted beside the answer key are searched for in the output (5) or the transcript (3); none has turned up. Blind judges never see the model or version; in some environments, transcripts are audited for off‑limits tools.

Judge agreement

In the 23 environments where both verdicts are recorded, the two judges disagreed on 994 of 16,613 judged results (6.0%): 3.5% in the office and household environments, 10.6% in the app builds and 9.0% in the server‑room incidents. Agreement isn't accuracy; no blind expert audit has been run yet.

What we know is limited

• Training evidence so far is one training run, on a simplified version of one environment; transfer to other environments isn't shown.
• Most environments have one attempt per model and version (46 of 56), so pass@k and the spread between attempts are measured on only 10.
• Many environments depend on LLM judges or LLM‑voiced people, and seven judge models are also models under test (Opus 5.5, Opus 4.8, Sonnet 5.5, GPT‑6.1 Sol, GPT‑5.6 Sol, GPT‑5.6 Luna, GPT‑6 Luna); Sonnet 5.5 also voices most simulated people. We haven't measured the reward under a second user simulator.
• In nine of the 12 app builds, 7% to 73% of check results carry a written ruling, many under each environment's written rules for rescoring; the other three don't record rulings per check.
• Costs are the agent model's cost at list prices only, and some are estimates where a model has no published price; the simulated people's and judges' calls aren't included, and costs weren't recorded for every attempt.
• 24 attempts on the 12‑month equipment plan, 7 of them with only the request, were scored against an earlier wording of the manager's records, with that wording's own best plan. GPT‑6 Astra's attempts on the temperature compliance register used an earlier build.
• Some shortcuts still score well. An empty course registration meets 11 of 14 checks in both course‑registration environments; a figures‑only report scores 84% in the waste report and a bare table 97% in the input‑price report; a lucky early flight about 90% in the stranded‑family rebooking.
• Doing nothing isn't 0 everywhere. Posting nothing scores 50% in the tea‑factory ops chat, and doing nothing 48% to 53% an act at the coffee order desk; a near‑empty attempt scores about 55% in the seasonal bulk order, and about 30% in the stranded‑family rebooking's self‑test.
• An idle agent earns 39% in the two‑site spares buying, 18% in the spares pick lists and about 2% in the materials planning and the drawing update, though their own graders fail it; one that hands in no report scores 10% in the server‑hall diagnosis.
• Leaving the starting registration meets 7 of 14 checks in both course‑registration environments; in the cancelled‑flight rebooking, a single read‑only lookup scores about 20%; the first seven app builds have no empty‑app test.
• No deliberate red‑team attempts or blind expert audit of the reward yet.
• In the two holiday‑rental environments, attempts weren't isolated from the answer key. At the coffee roaster's materials planning, Claude Code's 4 attempts ran outside a container, beside the simulator and grader, and their transcripts were checked afterwards; the 3 Codex CLI attempts ran in Harbor.
• Three environments are downloadable as Harbor tasks so far (the samples), tested with their reference solutions and empty attempts. A Claude agent has attempted two of them outside Harbor, graded by their own graders. We package the others on request.