Asapi / Methods
Methods
How the environments are built and graded
The words we use, how the worlds work, how we tested 28 models, how the reward is decided and checked, and what we know is limited.
The words on these pages
- Environment
- A simulated world (files, services, people, and a clock where time matters), a request, and a grader.
- Attempt
- One model's go at one environment, from the request to its last action, in one version of the request.
- Versions
- With only the request: as a user would give it. With a hint: plus a short pointer to what matters.
- Told the rules
- The request plus the rules or method the job depends on. In 11 of 42 environments it also gives facts from the files or people; in five of those, figures the grader checks.
- Variant
- A copy of an environment with different data, people and facts under the same design.
- Check and reward
- A check is one requirement the grader tests. The reward, from 0 to 1, is usually the weighted share of checks met; some environments use a cost model or a formula their datasheet sets out.
- Full solve
- An attempt that met every requirement, scoring 1.
- Gates and pass/fail
- Some environments cut or zero the reward when a make‑or‑break check fails. In 21, the grader also reports its own pass/fail. These pages use the reward; the samples report both.
- Capability
- Each environment is tagged with the capabilities its reward depends on, checked against its grader; 69 of 80 have two or more.
What's in each world
- Data
- Synthetic businesses and households with their own files: up to 4,490 at a time in mixed formats (16,554 over one environment's four acts), or case systems with tens of thousands of records. Planted facts, errors and look‑alikes are set by design and recorded in the answer key.
- Services
- Web apps, booking and supplier sites, banks, registration portals and simulators, each holding state the agent can change; every action is logged.
- Simulated people
- LLM‑voiced or scripted people who answer only what they're asked; the voice can run on a model you choose. See clarification with simulated users.
- Time
- Single sessions with a fixed in‑world date; in‑world clocks that move with each action; or simulated days, weeks and months in many sessions, each opened by mail or a schedule, with only the agent's own files carried between them.
- Checks before any attempt
- Self‑tests check that the answer key passes and the starting state fails; in some environments, gates prove that every deciding record can be found by reasonable searches.
How we tested 28 models
- Attempts
- 796 with only the request, 382 with a hint and 400 told the rules, plus 1 in a fourth version of one environment.
- Models
- 28 models from eight makers, 1 to 16 per environment.
- OpenAI
- GPT‑5.6 Luna, GPT‑5.6 Sol, GPT‑5.6 Terra, GPT‑6 Astra, GPT‑6 Luna, GPT‑6 Sol, GPT‑6.1 Sol, GPT‑6.1 Terra.
- Anthropic
- Fable 5.1, Fable 5.2, Haiku 4.5, Haiku 5.5, Opus 4.8, Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5.
- Gemini 4 Argon.
- Open‑weight
- Alibaba (Qwen3‑235B, Qwen3.8‑27B); DeepSeek (DeepSeek V3.2, DeepSeek V4.1 Flash); Moonshot (Kimi K2.5, Kimi K3); NVIDIA (Nemotron Super 3); Zhipu (GLM‑5, GLM‑5.3, GLM‑5.3‑Flash).
- Harnesses
- Each model ran in an agent harness: OpenAI's in Codex CLI, almost all of Anthropic's in Claude Code, Google's in Antigravity CLI, and open‑weight models in Kimi CLI, Qwen Code and other harnesses. Most models used one harness throughout, so model and harness effects are mixed.
- Sandboxes
- Most attempts ran in a Docker container; in 26 environments, some or all ran locally, in a macOS sandbox, a plain working folder or, for the 11 robot jobs and the two races, on a Windows PC, and the two course registrations use a portal harness.
- Budgets
- Each attempt had limits set per environment, on wall‑clock time and usually on turns or spend; for example, 150 turns, 120 minutes and $10–20 for the app builds.
- Versions
- With only the request on all 80; with a hint on 34; told the rules on 42.
- Repeats
- One attempt per model and version on 65 environments. Repeated attempts on 15: three per model, except at the coffee roaster's materials planning, where models made one or two, and in the condo floor repair, where one model made two told the rules.
- When
- September to 18 November 2026, so the results are for the model versions available then.
Scripts first, judges where needed
- Script checks
- Pooled over attempts with only the request, code decides about nine in ten check results and most points. In about a dozen environments, judges decide most points.
- What scripts check
- The final state of bookings, payments and registrations; documents field by field, or byte for byte where the format is fixed; figures and dates exactly.
- Hidden tests
- The two spec‑to‑code environments run hidden test suites (466 and 919 tests) on inputs the agent never sees.
- Simulators
- Most long‑horizon environments replay the action log through the simulator after the last session and check that the replay reproduces the recorded run.
- Cost models
- Five environments price the plan. Two score from a baseline (0) to the best of tens of thousands of candidates (1), times the share of key facts quoted. Profit is 70% of the bulk order's reward; the design's value is half the rooftop quote's and scales the factory solar's.
- Tester models
- App builds are operated by a tester model in a real browser, limited to page actions, while scripts probe security, restarts, recovery and clock changes.
- LLM judges
- Where a script can't decide, judges answer fixed yes/no questions: usually two from different model families (for example Opus 4.8 and GPT‑5.6 Luna), or two blind readers with reference and near‑miss verdicts mixed into every batch. Both must agree for a full pass. Judges can run on models you choose.
- No judge
- 44 of 80 environments are graded by code alone.
- Rulings
- In the bakery, brewery and household environments, the server‑room cooling failure and the lighting plan, a judge split pays half. Elsewhere a written ruling settles it: by the environment's author, Claude for the author, a coordinator, Opus 5.5 in three app builds, or the grading session.
Checks on the reward
- Difficulty
- With only the request, 47 of 796 attempts (5.9%) were full solves and 93% earned some reward; the middle half of attempts scored 32–86%.
- Where full solves happen
- 45 of 218 attempts on the 38 environments without a told‑the‑rules version, mostly long‑horizon and coding; 2 of 578 on the other 42.
- Solvable
- In the 34 environments where the same models were tested on both, full solves rose from 2 to 64 of 383 attempts once told the rules, and the median reward from 56% to 88%. In the 26 whose told‑the‑rules version gives no facts: 1 to 33 of 257.
- Best when told the rules
- The best attempt told the rules reaches 95% or more on 28 of 42 environments, and a full solve on 13. Eight of the fourteen below 95% are app builds.
- Training run
- DeepSeek 14B, trained on a simplified version of one environment, went from 20 to 97 out of 100, out of sample. It is the one training run so far.
- Right for the right reason
- In 9 environments, a correct decision earns full credit only if the deciding fact was in the attempt's log before it was made.
- Doing nothing
- In at least 38 environments the grader's own self‑test requires that doing nothing scores 0, and checks that a mistake wasn't made pass only for attempts that did the work. The exceptions are under limitations.
- Reference solutions
- They fully solve 51 of 80. The 11 robot jobs, the two races and 10 of 12 app builds have none; two app builds, the server‑room cooling failure and the lighting plan have one graded only in part. In two long jobs the reference passes but misses a few checks.
- Gated packages
- In 24, the package is gated: the reference must pass and doing nothing must fail.
- Shortcuts
- We graded degenerate strategies in 79 of 80. Most score little: last month's report copied forward scores under 5%, and returning every listing scores 0.
- Answer hunting
- In our attempts' logs, one model, DeepSeek V4.1 Flash, searched its container, a simulator's service and the web for answers or the answer key in several attempts. It found nothing useful, and its scores stand.
- Leak checks
- In 9 environments, canary strings planted beside the answer key are searched for in the output (6) or the transcript (3); none has turned up. Blind judges never see the model or version; in some environments, transcripts are audited for off‑limits tools.
- Judge agreement
- Before the 17 and 18 November regrades, in the 25 environments with both verdicts recorded, the judges disagreed on 6.2% of judged results: 3.7% in office and household environments, 10.6% in app builds, 9.0% in server‑room incidents and 11.8% in the lighting plan and condo floor repair.
- Agreement isn't accuracy
- No blind expert audit has been run yet.
What we know is limited
- Training evidence so far is one training run, on a simplified version of one environment; transfer to other environments isn't shown.
- Most environments have one attempt per model and version (65 of 80), so pass@k and the spread between attempts are measured on only 15.
- Many environments depend on LLM judges or LLM‑voiced people, and seven judge models are also models under test (Opus 5.5, Opus 4.8, Sonnet 5.5, GPT‑6.1 Sol, GPT‑5.6 Sol, GPT‑5.6 Luna, GPT‑6 Luna); Sonnet 5.5 also voices most simulated people. We haven't measured the reward under a second user simulator.
- In nine of the 12 app builds, 7% to 73% of check results carry a written ruling, many under each environment's written rules for rescoring; the other three don't record rulings per check.
- Costs are the agent model's cost at list prices only, and some are estimates where a model has no published price; the simulated people's and judges' calls aren't included, and costs weren't recorded for every attempt.
- 24 attempts on the 12‑month equipment plan, 7 of them with only the request, were scored against an earlier wording of the manager's records, with that wording's own best plan. GPT‑6 Astra's attempts on the temperature compliance register used an earlier build.
- Some shortcuts still score well. An empty course registration meets 11 of 14 checks in both course‑registration environments; a bare table with none of the asked‑for notes scores 49% in the waste report and 64% in the input‑price report; a lucky early flight about 90% in the stranded‑family rebooking.
- Doing nothing isn't 0 everywhere: a near‑empty attempt scores about 42% in the bulk order, about 30% in the stranded‑family rebooking's self‑test and 5% in the family's move.
- An idle agent earns 12% in the plant watchdog and about 2% in the materials planning and the drawing update, though their own graders fail it.
- Leaving the starting registration meets 7 of 14 checks in both course‑registration environments; in the cancelled‑flight rebooking, a single read‑only lookup scores about 20%; the first seven app builds have no empty‑app test.
- No deliberate red‑team attempts or blind expert audit of the reward yet.
- At the coffee roaster's materials planning, Claude Code's 4 attempts ran outside a container, beside the simulator and grader, and their transcripts were checked afterwards; the 3 Codex CLI attempts ran in Harbor.
- In the two course registrations, each Codex CLI attempt ran in a read‑only sandbox that could still read the harness's files, its reference plan among them; its tool calls weren't kept.
- No reference solution or attempt has completed any of the ten robot builds, and the batting job has no reference run, so their solvability is unproven; their targets and engine are open to the agent by design, with limits on reading the grader set by instruction only.
- Three environments are downloadable as Harbor tasks so far (the samples), tested with their reference solutions and empty attempts. A Claude agent has attempted two of them outside Harbor, graded by their own graders. We package the others on request; the robot jobs haven't been tried in Harbor.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials