Asapi
Founder, CEO
Dr. Qingchen Wang
- VC‑backed founder
- Kaggle Grandmaster
- Former University of Hong Kong professor
For AI labs
RL environments for agentic work
56 environments where an agent does a whole job, and a grader turns the result into a reward. For example:
- Approve a contractor's invoice from 171 messy files
- Coordinate care on a hospital ward for three simulated days
- Organise a family dinner for 40 by messaging relatives and restaurants
Environments, variants and custom builds
The environments
Each one's files, tools and grader, with its simulated people and clock where it has them. People are made up; two use anonymised rental listings.
In your stack
Three are downloadable as Harbor tasks, a format SkyRL, Tinker and NeMo Gym read; we've tested them in Harbor's runner only. Request one.
Fresh variants
New, unpublished instances of any environment, as many as you need.
Baseline results
Our test attempts on each environment, with the grader's report on each.
Custom builds
Environments for your domain, tools and users, graded the same way.
Licences and pricing
Exclusive or non‑exclusive, at launch pricing; samples go out on request.
People, long jobs, and credit for the reason
People who answer only what's asked
In 9, code releases each fact only to a question asking for it; the voice model words only released lines.
Long jobs, with only the agent's notes
In 10 environments, the agent works through simulated days, weeks or months, keeping only its own files between sessions.
Credit for the reason, not the guess
In 5, a right decision earns full credit only if the agent found the deciding fact first; a lucky guess earns less.
Hard for today's models, and solvable
We tested 23 models, each given only the request, as a user would.
42 of 56Environments that none of the models we tested fully solved (a median of 7.5 models tested on each)
64Requirements checked in the median environment, each earning part of the reward
48 of 250Attempts that fully solved the job once told the rules, against 2 with only the request (the same models, 25 environments)
A job's rules and facts sit in files or with people. Told the rules: given the rules or method up front, no facts.
Five safeguards on the reward
Each covers some environments; Methods lists which.
- Code decides most of it
- Scripts decide about 84% of points, 78% averaged across environments, from the final state and files; LLM judges decide the rest.
- Answers kept out of reach
- Answer keys stay out of reach, except the holiday‑rental pair and Claude Code's coffee‑roaster attempts; 10 run the agent's program on unseen inputs.
- Empty and reference attempts
- In 25, a self‑test requires an empty attempt to score 0. Reference solutions fully solve 42 of 56; 10 of 12 app builds have none.
- Judges on fixed questions
- Judges answer fixed yes/no questions, on your models. Written rulings settled our splits; Methods says who wrote them.
- Leak checks
- In 8 environments, we search output or transcripts for planted canary strings; none has turned up. One model hunted for answers and found nothing useful.
Partial credit is common; full solves are rare
Each dot is one attempt, with only the request; the line marks the median. Larger dots are full solves: every requirement met.
ModelMedianFully solved
Opus 5.5Anthropic · 55 environments85%11 of 74
GPT‑6 AstraOpenAI · 53 environments82%8 of 71
GPT‑6.1 SolOpenAI · 52 environments80%6 of 70
Fable 5.1Anthropic · 45 environments78%5 of 63
Sonnet 5.5Anthropic · 53 environments71%3 of 72
GPT‑6 LunaOpenAI · 52 environments31%0 of 70
The six models tested on at least 40 of the 56, each a different subset.
Ten capabilities, model by model
Each model's median reward on a capability's environments, and its full solves out of its attempts there. Attempts count toward every capability their environment tests.
| Capability | Opus 5.5 | GPT‑6 Astra | GPT‑6.1 Sol | Fable 5.1 | Sonnet 5.5 | GPT‑6 Luna |
|---|---|---|---|---|---|---|
| Agentic coding15 environments | 84%3 of 15 solved | 75%1 of 15 solved | 66%1 of 15 solved | 73%1 of 14 solved | 74%1 of 15 solved | 19%0 of 15 solved |
| Stateful tool use19 environments | 92%8 of 27 solved | 77%4 of 24 solved | 79%4 of 23 solved | 74%3 of 20 solved | 64%1 of 26 solved | 25%0 of 24 solved |
| Clarifying with people17 environments | 81%1 of 25 solved | 83%3 of 25 solved | 79%1 of 25 solved | 77%1 of 22 solved | 69%1 of 25 solved | 22%0 of 24 solved |
| Planning ahead18 environments | 90%7 of 24 solved | 78%5 of 22 solved | 78%4 of 21 solved | 74%3 of 17 solved | 64%1 of 24 solved | 32%0 of 22 solved |
| Multimodal records21 environments | 81%1 of 29 solved | 83%1 of 29 solved | 79%0 of 29 solved | 83%1 of 27 solved | 75%2 of 29 solved | 21%0 of 28 solved |
| Unstated requirements36 environments | 83%4 of 49 solved | 82%2 of 48 solved | 78%1 of 48 solved | 78%0 of 46 solved | 72%2 of 47 solved | 31%0 of 47 solved |
| Catching errors33 environments | 83%1 of 46 solved | 83%1 of 45 solved | 81%0 of 44 solved | 80%1 of 41 solved | 74%1 of 44 solved | 35%0 of 44 solved |
| Disclosure and honesty16 environments | 85%0 of 28 solved | 83%0 of 28 solved | 82%0 of 28 solved | 83%0 of 28 solved | 73%0 of 28 solved | 31%0 of 28 solved |
| When to act or escalate26 environments | 88%5 of 39 solved | 84%2 of 37 solved | 84%1 of 36 solved | 83%1 of 29 solved | 71%1 of 38 solved | 31%0 of 36 solved |
| Long‑horizon work10 environments | 100%7 of 13 solved | 94%4 of 11 solved | 97%4 of 10 solved | 100%3 of 4 solved | 92%1 of 12 solved | 82%0 of 11 solved |
A dash: not tested on any of these environments.
What the models get wrong
One example per capability, counting every model tested on the environment, each given only the request.
- Long‑horizon work1 of 2
Attempts serviced worn trucks only after faults and kept using a reported potholed road: 8 breakdowns in a week, against none for the other
- Clarifying with people7 of 9
Attempts took a relative's second‑hand word that a household abroad wasn't coming, and never asked the household
- Stateful tool use4 of 4
Attempts kept a course a registration portal accepted, though the student lacked its prerequisite
- Agentic coding8 of 8
Apps built for a home‑loan broker put the broker's client identifiers in their code or tests
- Multimodal records8 of 13
Attempts never opened the maker's product sheet, an image inside a PDF that gave a part's true size
- Catching errors11 of 14
Attempts relied on a later, edited copy of a key record that same‑day evidence contradicts
- Disclosure and honesty5 of 8
Attempts' final notes told the owner something had happened that hadn't
- Planning ahead8 of 13
Attempts ordered the new cabinet asked for, scoring 0; those that first found the old one's fault and acted scored 44–100%
- When to act or escalate3 of 6
Attempts on a hospital ward didn't page a doctor in time about a dose charted wrong on the third day
- Unstated requirements12 of 13
Attempts kept an account the owner's latest rule excludes, following last month's pack instead
One training run so far
DeepSeek 14B, trained on a simplified version of one environment. Score out of 100, out of sample:
It shows the simplified environment can be learned. Transfer to other environments isn't tested yet.
What isn't built or measured yet
- Some shortcuts still pay: a bare table scores 97% in one report environment. There are no red‑team attempts yet.
- On 46 of the 56 environments each model made one attempt, so pass@k (success within k attempts) is measured on only 10.
- We haven't measured how reward changes when other models judge or voice the simulated people.
Who we are
Machine learning researchers with a combined 60+ years of experience.
Founding advisor
Dr. Stefano Ermon
- Stanford professor
- Diffusion models inventor
- Inception Labs founder
Team affiliations
Try a sample environment
Three are downloadable as Harbor tasks, with graders and reference solutions. Request one with your work email, or book a call about the other 53.