Long‑horizon work
Some jobs run for days, weeks or months, in sessions that start with no memory of the last. Doing them well means keeping notes the agent can rely on, following through on what it started, and leaving things running safely.
Three things an agent has to get right
Keep notes it can rely on
Each session starts fresh, and only the agent's own files carry over, so its notes must hold whatever the next session needs.
Follow through
Promises, orders and plans made days or weeks earlier still bind, and handoffs between shifts must carry every piece of pending work.
Leave things running safely
Some environments have the agent leave programs, scheduled jobs or watchers running between its sessions, even through a reboot.
Their notes fail them, threads get dropped, and programs go unchecked
Notes that let it down
Packing‑line project purchasing: in 17% of attempts, the agent spread its notes over many files that later sessions often didn't read, so settled questions were decided again; none of those attempts passed.
Coffee roaster materials planning: in 29% of attempts, a material the agent had stopped tracking ran dry, such as cartons its own notes said needed topping up, and a customer's exit clause ended the season early.
Promises and handoffs not kept
Month‑long parcel fleet dispatch: in 25% of attempts, the agent planned a day from the new parcels alone, leaving off ones it had already promised, and two missed days in a row ended the month.
Three‑day hospital ward care coordination: only 33% of attempts got all seven shift handoffs exactly right; one carried an expired one‑night dose change forward as still due.
Programs left unchecked
Watchdog program for a running plant: in 50% of attempts, the watchdog the agent left mishandled the plant after a leak: it stopped acting once it had raised the alarm, or kept a line shut while the crew restored it.
Coffee web‑shop order desk: in 20% of attempts, a scheduled job the agent wrote kept shipping trade orders days early between sessions, unnoticed until finance took over the desk.
Most attempts earn most of the reward; about half the environments have a full solve
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Notes. Over three sessions in five weeks, later sessions see the phones' sensor data only as the agent's own code kept it. | 81 | 5 | 98% |
| Month‑long parcel fleet dispatch, notes‑only memorySmall business | Follow‑through. Fresh sessions over a simulated month keep only the agent's files, and each day's promises and truck wear carry into the next. | 29 | 4 | 100% |
| Tea‑factory cash desk over thirteen weekly sessionsSmall business | Notes. Thirteen weekly sessions keep only the agent's files, and buyers' prices have to be re‑learned from accepted and refused quotes each week. | 64 | 5 | 100% |
| Two‑site spares pick lists and stock plansManufacturing | Follow‑through. Nine fresh sessions keep only the agent's files, and the stock plan, refiled week by week, is checked at weeks 4 and 8. | 35 | 6 | 100% |
| Packing‑line project purchasing over twelve mail‑driven weeksManufacturing | Notes. Dozens of sessions over twelve weeks keep only the agent's files, and orders placed weeks earlier decide what reaches site in time. | 14 | 6 | 100% |
| Two‑site spares buying through supplier part withdrawalsManufacturing | Follow‑through. Dozens of sessions over twelve weeks keep only the agent's files, with several supplier threads open at once. | 62 | 5 | 100% |
| Coffee roaster materials planning across two seasonsManufacturing | Follow‑through. Each 16‑week season runs up to 80 fresh sessions keeping only the agent's files, and a customer let down too often ends it early. | 112–166 | 5 | 98% |
| Tea‑factory ops chat: speak or stay silentSmall business | Notes. Weeks of chat must fit into 16 KB of notes, and later checks have only those, the guide and the new messages. | 71 | 6 | 53% |
| Coffee web‑shop order desk through a four‑act Christmas peakSmall business | Unattended. Up to 48 sessions per act keep only the agent's files, and the paperwork runs through jobs the agent schedules between sessions. | 207–213 | 5 | 94% |
| Three‑day hospital ward care coordination, notes‑only memoryHospital | Follow‑through. Fresh sessions over 72 simulated hours keep only the agent's notes, and pending work must pass across shifts in exact handoffs. | 290–415 | 6 | 100% |
| Watchdog program commissioned for a running solvent recovery unitProcess control | Unattended. After its session ends, the agent's program runs the plant alone for a day, through a reboot, keeping only its own files. | 41 | 6 | 93% |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials