AsapiThe AI Agent Trials

Training environments for agents that do real work

We turn real jobs into reinforcement‑learning environments. An agent works through a whole task in a simulated business or household, with files, tools and sometimes people to talk to, and a grader turns what it hands in into a reward. We've built 80 so far, for example:

  • Approve a contractor's invoice from 171 messy files
  • Coordinate care on a hospital ward for three simulated days
  • Organise a family dinner for 40 by messaging relatives and restaurants
FILESTOOLSPEOPLECLOCKAIAGENT IN A CONTAINERGRADER01REWARD
Why these environments

Built around what models still miss in real work

Real jobs, with the details left out

Many are jobs someone might hand to an assistant, asked the way they'd ask. In most, what the job really needs sits in files, records or people's heads, so the agent has to find it before it acts.

People to talk to, and time that passes

In some environments the agent has to ask people for what it needs, and they answer only what they're asked, as busy colleagues do. In others, the work runs over simulated days, weeks or months, and the agent keeps only its own files between sessions.

A reward that pays for partial progress

In most environments, each of dozens of requirements earns part of the reward, so an agent earns credit for what it gets right. Some also have make‑or‑break requirements, and in some a right decision earns full credit only if the agent first found the fact that decides it.

What we found

Today's most powerful models get most of the way, but still make serious mistakes

We tested 28 models on these environments, each given only the request, with no hints and no rules spelled out.

70%The median score across every attempt
94%Attempts that didn't completely solve the job

Each dot is one attempt, and the line marks a model's median. The strongest models typically earn around four‑fifths of the reward, and fully solve about one attempt in ten.

ModelMedianFully solved
Opus 5.5Anthropic · 67 environments83%11%
GPT‑6 AstraOpenAI · 69 environments83%10%
Fable 5.1Anthropic · 50 environments78%6%
GPT‑6.1 SolOpenAI · 78 environments76%9%
Sonnet 5.5Anthropic · 65 environments71%4%
GPT‑6 LunaOpenAI · 69 environments26%1%

The six models we've tested on at least 40 environments.

Share of attempts that made each mistake:

  1. Family dinner

    78% of attempts

    Never checked a rumour that one family wasn't coming, and booked too few seats

  2. Home‑loan broker's app

    100% of attempts

    Left the broker's client IDs in the app's code or tests

  3. Hospital ward

    50% of attempts

    Didn't page a doctor in time about a wrongly charted dose

  4. Ice cream cabinet

    69% of attempts

    Bought a new cabinet when a cheaper repair would fix the old one

Capabilities

Thirteen capabilities the environments target

Each environment targets several. A capability's page lists its environments and how each model did.

The reward

A reward you can check

Across the environments, code decides most of the reward, by reading the final state of the simulated world and the files the agent leaves behind. Where a judgment is needed, such as whether a letter says the right thing, AI judges answer fixed yes‑or‑no questions.

The three samples you can download show how we package an environment. The grader and its answer key sit outside the agent's reach, a reference solution earns the whole reward, an empty attempt earns nothing, and any judge or model‑voiced person runs on whichever model you choose.

WORLD STATEFILESGRADERCODEAI JUDGE?YESNO01REWARDEMPTY ATTEMPTREFERENCE SOLUTION
What you get

Environments, fresh variants and custom builds

The environments

Each with its files, tools and grader, and its simulated people and clock where it has them.

Fresh variants

New, unpublished instances of an environment, as many as you need, so you can keep some back for evaluation.

Custom builds

Environments for your own domain, tools and users, graded the same way.

Our test results

The attempts we made on each environment, with the grader's report on each scored one, to compare yours against.

In your training stack

The samples come as Harbor tasks, an open format for agent tasks, and we package other environments the same way on request.

Licences

Exclusive or non‑exclusive, at launch prices.

Training

A first training run

We trained a 14B DeepSeek model on a simplified version of one environment. On instances it hadn't seen, its score went from 20 to 97 out of 100.

Before training20
After training97
The team

Who we are

Machine learning researchers with a combined 60+ years of experience.

Dr. Qingchen Wang

Founder, CEO

Dr. Qingchen Wang

  • VC‑backed founder
  • Kaggle Grandmaster
  • Former University of Hong Kong professor
Dr. Stefano Ermon

Founding advisor

Dr. Stefano Ermon

  • Stanford professor
  • Diffusion models inventor
  • Inception Labs founder
Team affiliations
UCLUniversity of TorontoUniversity of WaterlooUniversity of EdinburghNorthwestern UniversityHarvard UniversityStanford UniversityCarnegie Mellon UniversityGoogleMetaTuringUniversity of Oxford

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.