AsapiThe AI Agent Trials
Asapi / Methods
Methods

How the environments are built and graded

The words we use, how the worlds work, how we tested 28 models, how the reward is decided and checked, and what we know is limited.

FILESTOOLSPEOPLECLOCKAIAGENT IN A CONTAINERGRADER01REWARD
Definitions

The words on these pages

Environment
A simulated world (files, services, people, and a clock where time matters), a request, and a grader.
Attempt
One model's go at one environment, from the request to its last action, in one version of the request.
Versions
With only the request: as a user would give it. With a hint: plus a short pointer to what matters.
Told the rules
The request plus the rules or method the job depends on. In 11 of 42 environments it also gives facts from the files or people; in five of those, figures the grader checks.
Variant
A copy of an environment with different data, people and facts under the same design.
Check and reward
A check is one requirement the grader tests. The reward, from 0 to 1, is usually the weighted share of checks met; some environments use a cost model or a formula their datasheet sets out.
Full solve
An attempt that met every requirement, scoring 1.
Gates and pass/fail
Some environments cut or zero the reward when a make‑or‑break check fails. In 21, the grader also reports its own pass/fail. These pages use the reward; the samples report both.
Capability
Each environment is tagged with the capabilities its reward depends on, checked against its grader; 69 of 80 have two or more.
Worlds

What's in each world

Data
Synthetic businesses and households with their own files: up to 4,490 at a time in mixed formats (16,554 over one environment's four acts), or case systems with tens of thousands of records. Planted facts, errors and look‑alikes are set by design and recorded in the answer key.
Services
Web apps, booking and supplier sites, banks, registration portals and simulators, each holding state the agent can change; every action is logged.
Simulated people
LLM‑voiced or scripted people who answer only what they're asked; the voice can run on a model you choose. See clarification with simulated users.
Time
Single sessions with a fixed in‑world date; in‑world clocks that move with each action; or simulated days, weeks and months in many sessions, each opened by mail or a schedule, with only the agent's own files carried between them.
Checks before any attempt
Self‑tests check that the answer key passes and the starting state fails; in some environments, gates prove that every deciding record can be found by reasonable searches.
Our attempts

How we tested 28 models

Attempts
796 with only the request, 382 with a hint and 400 told the rules, plus 1 in a fourth version of one environment.
Models
28 models from eight makers, 1 to 16 per environment.
OpenAI
GPT‑5.6 Luna, GPT‑5.6 Sol, GPT‑5.6 Terra, GPT‑6 Astra, GPT‑6 Luna, GPT‑6 Sol, GPT‑6.1 Sol, GPT‑6.1 Terra.
Anthropic
Fable 5.1, Fable 5.2, Haiku 4.5, Haiku 5.5, Opus 4.8, Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5.
Google
Gemini 4 Argon.
Open‑weight
Alibaba (Qwen3‑235B, Qwen3.8‑27B); DeepSeek (DeepSeek V3.2, DeepSeek V4.1 Flash); Moonshot (Kimi K2.5, Kimi K3); NVIDIA (Nemotron Super 3); Zhipu (GLM‑5, GLM‑5.3, GLM‑5.3‑Flash).
Harnesses
Each model ran in an agent harness: OpenAI's in Codex CLI, almost all of Anthropic's in Claude Code, Google's in Antigravity CLI, and open‑weight models in Kimi CLI, Qwen Code and other harnesses. Most models used one harness throughout, so model and harness effects are mixed.
Sandboxes
Most attempts ran in a Docker container; in 26 environments, some or all ran locally, in a macOS sandbox, a plain working folder or, for the 11 robot jobs and the two races, on a Windows PC, and the two course registrations use a portal harness.
Budgets
Each attempt had limits set per environment, on wall‑clock time and usually on turns or spend; for example, 150 turns, 120 minutes and $10–20 for the app builds.
Versions
With only the request on all 80; with a hint on 34; told the rules on 42.
Repeats
One attempt per model and version on 65 environments. Repeated attempts on 15: three per model, except at the coffee roaster's materials planning, where models made one or two, and in the condo floor repair, where one model made two told the rules.
When
September to 18 November 2026, so the results are for the model versions available then.
Grading

Scripts first, judges where needed

Script checks
Pooled over attempts with only the request, code decides about nine in ten check results and most points. In about a dozen environments, judges decide most points.
What scripts check
The final state of bookings, payments and registrations; documents field by field, or byte for byte where the format is fixed; figures and dates exactly.
Hidden tests
The two spec‑to‑code environments run hidden test suites (466 and 919 tests) on inputs the agent never sees.
Simulators
Most long‑horizon environments replay the action log through the simulator after the last session and check that the replay reproduces the recorded run.
Cost models
Five environments price the plan. Two score from a baseline (0) to the best of tens of thousands of candidates (1), times the share of key facts quoted. Profit is 70% of the bulk order's reward; the design's value is half the rooftop quote's and scales the factory solar's.
Tester models
App builds are operated by a tester model in a real browser, limited to page actions, while scripts probe security, restarts, recovery and clock changes.
LLM judges
Where a script can't decide, judges answer fixed yes/no questions: usually two from different model families (for example Opus 4.8 and GPT‑5.6 Luna), or two blind readers with reference and near‑miss verdicts mixed into every batch. Both must agree for a full pass. Judges can run on models you choose.
No judge
44 of 80 environments are graded by code alone.
Rulings
In the bakery, brewery and household environments, the server‑room cooling failure and the lighting plan, a judge split pays half. Elsewhere a written ruling settles it: by the environment's author, Claude for the author, a coordinator, Opus 5.5 in three app builds, or the grading session.
Validation

Checks on the reward

Difficulty
With only the request, 47 of 796 attempts (5.9%) were full solves and 93% earned some reward; the middle half of attempts scored 32–86%.
Where full solves happen
45 of 218 attempts on the 38 environments without a told‑the‑rules version, mostly long‑horizon and coding; 2 of 578 on the other 42.
Solvable
In the 34 environments where the same models were tested on both, full solves rose from 2 to 64 of 383 attempts once told the rules, and the median reward from 56% to 88%. In the 26 whose told‑the‑rules version gives no facts: 1 to 33 of 257.
Best when told the rules
The best attempt told the rules reaches 95% or more on 28 of 42 environments, and a full solve on 13. Eight of the fourteen below 95% are app builds.
Training run
DeepSeek 14B, trained on a simplified version of one environment, went from 20 to 97 out of 100, out of sample. It is the one training run so far.
Right for the right reason
In 9 environments, a correct decision earns full credit only if the deciding fact was in the attempt's log before it was made.
Doing nothing
In at least 38 environments the grader's own self‑test requires that doing nothing scores 0, and checks that a mistake wasn't made pass only for attempts that did the work. The exceptions are under limitations.
Reference solutions
They fully solve 51 of 80. The 11 robot jobs, the two races and 10 of 12 app builds have none; two app builds, the server‑room cooling failure and the lighting plan have one graded only in part. In two long jobs the reference passes but misses a few checks.
Gated packages
In 24, the package is gated: the reference must pass and doing nothing must fail.
Shortcuts
We graded degenerate strategies in 79 of 80. Most score little: last month's report copied forward scores under 5%, and returning every listing scores 0.
Answer hunting
In our attempts' logs, one model, DeepSeek V4.1 Flash, searched its container, a simulator's service and the web for answers or the answer key in several attempts. It found nothing useful, and its scores stand.
Leak checks
In 9 environments, canary strings planted beside the answer key are searched for in the output (6) or the transcript (3); none has turned up. Blind judges never see the model or version; in some environments, transcripts are audited for off‑limits tools.
Judge agreement
Before the 17 and 18 November regrades, in the 25 environments with both verdicts recorded, the judges disagreed on 6.2% of judged results: 3.7% in office and household environments, 10.6% in app builds, 9.0% in server‑room incidents and 11.8% in the lighting plan and condo floor repair.
Agreement isn't accuracy
No blind expert audit has been run yet.
Known limitations

What we know is limited

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.