AsapiThe AI Agent Trials
Asapi / Capabilities / Stateful tool use
Capability · 25 environments

Stateful tool use

Agents often work through tools on systems that remember: every booking, order or message changes what they'll find next. Doing that well means keeping track of what the system holds, checking what each action did, and acting in time.

What it takes

Three things an agent has to get right

Keep track of what the system holds

Stock, bookings, payments and promises carry over from one step, or one week, to the next, and some change on their own.

Check what each action did

Some systems accept a mistake without a word, and others refuse an order without saying why, so the agent has to look for itself.

Act in order, and in time

Seats sell out, holds expire, purchases charge at once and an uncooled server room heats by the minute, so timing and order matter.

What models get wrong

They lose track, skip the check, or act too late

Losing track

  1. Coffee roaster materials planning: in 29% of attempts, a material the agent had stopped tracking ran dry, and a customer's exit clause ended the season early.

  2. Month‑long parcel fleet dispatch: in 25% of attempts, the agent planned a day from the new parcels alone, leaving off ones it had already promised, and two missed days in a row ended the month.

Not checking

  1. Course registration against rules found in portal: 69% of attempts didn't catch that the portal had silently accepted a class the student wasn't yet allowed to take.

  2. Coffee web‑shop order desk: in 20% of attempts, a scheduled job the agent wrote kept shipping trade orders days early, and the agent never checked its records until finance took over the desk.

Too late, or in the wrong order

  1. Cancelled‑flight family rebooking: in 58% of attempts, the no‑cost seats sold out before the agent booked them, and each of those attempts then spent between $680 and $1,070, after refunds, on other tickets.

  2. Server‑room cooling outage: in every attempt, the agent made calls or chased the cause before anything else, and the servers shut down from heat. A scripted reference plan keeps them running.

Results

Partial credit for most attempts, a full solve for few

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

EnvironmentMedianFully solved
Family dinner booking with simulated relatives12 models × 3 attempts10%0 of 36Cancelled‑flight family rebooking against a ticking clock8 models × 3 attempts70%0 of 24Course registration against rules found in portal16 models71%2 of 16Course registration optimised for earliest graduation15 models79%0 of 15Stranded‑family rebooking with minors' exit papers and a dog8 models × 3 attempts61%0 of 24Seasonal bulk order run with simulated staff8 models46%0 of 8Unattended train‑ticket buyer for a rate‑limited booking site8 models83%1 of 8Smart‑home retrofit: purchase plan, wiring and local control software7 models76%0 of 7Month‑long parcel fleet dispatch, notes‑only memory4 models62%2 of 4Tea‑factory cash desk over thirteen weekly sessions5 models81%1 of 5Two‑site spares pick lists and stock plans6 models90%1 of 6Packing‑line project purchasing over twelve mail‑driven weeks6 models × 3 attempts100%13 of 18Two‑site spares buying through supplier part withdrawals5 models97%2 of 5Coffee roaster materials planning across two seasons5 models, 1 or 2 attempts each90%0 of 7Coffee web‑shop order desk through a four‑act Christmas peak5 models90%0 of 5Three‑day hospital ward care coordination, notes‑only memory6 models95%1 of 6Server‑room cooling outage in a 3D building with model‑played staff7 models5%0 of 7Server‑hall cooling diagnosis by undoable tests in a 3D building7 models35%0 of 7800 kg air handler move proved by trial in a 3D physics model7 models33%0 of 7Watchdog program commissioned for a running solvent recovery unit6 models79%0 of 6Room furnishing from photos: measure, match look‑alikes, order and plan8 models × 3 attempts63%0 of 24Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up8 models × 3 attempts75%0 of 24Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layouts8 models × 3 attempts83%0 of 24Night repair of a leaking condo floor on an in‑world clock7 models × 3 attempts41%1 of 21Club race‑car design and parts procurement on a web app with hidden supplier terms5 models96%0 of 5

Hover a dot for its model and reward.

The 25 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Family dinner booking with simulated relativesHouseholdChecking. Holds expire, payments land hours later and one confirmation carries an error until queried, so every booking and payment needs checking.931293%
Cancelled‑flight family rebooking against a ticking clockTravel conciergeIn time. Pages, forms and messages move the clock, and seats sell out, so the no‑cost seats have to be secured before they're gone.61–63885%
Course registration against rules found in portalUniversity portalChecking. The portal registers whatever it's given, so the agent has to check the final timetable against rules set out on its own pages.1416100%
Course registration optimised for earliest graduationUniversity portalChecking. The same unguarded portal, where the agent has to leave the registration that keeps every rule and graduates the student soonest.141593%
Stranded‑family rebooking with minors' exit papers and a dogPersonal assistantKeeping track. Changes to flights, the car, the dog's travel and claims spread across a hub and nine sites, and must all fit together.34896%
Seasonal bulk order run with simulated staffSmall businessKeeping track. Figures confirmed with the customer must match bakes, staff hours and bookings week by week; profit comes from where they end up.23875%
Unattended train‑ticket buyer for a rate‑limited booking siteHouseholdIn time. The agent's app has to watch a site with its own clock, where returned seats come back briefly and holds expire.268100%
Smart‑home retrofit: purchase plan, wiring and local control softwareHouseholdChecking. The agent's software runs a house's devices over their real protocols and has to show their true state, even when one goes offline.121792%
Month‑long parcel fleet dispatch, notes‑only memorySmall businessKeeping track. Promises, truck wear and repairs carry over day to day, so each day's plan has to start from what's actually due.294100%
Tea‑factory cash desk over thirteen weekly sessionsSmall businessIn time. Each week's sales move the books and bank balance the next week starts from, and the owner's money takes three weeks to arrive.645100%
Two‑site spares pick lists and stock plansManufacturingKeeping track. Stock moves as technicians take parts, and the stock plan is tested twice against the downtime limit at both sites.356100%
Packing‑line project purchasing over twelve mail‑driven weeksManufacturingIn time. Orders placed over twelve weeks have to land before the installation step that needs them, though one seller's real lead times run long.146100%
Two‑site spares buying through supplier part withdrawalsManufacturingIn time. Suppliers withdraw parts, and replacements have to be in stock at both sites before a repair waits; orders can be refused unexplained.625100%
Coffee roaster materials planning across two seasonsManufacturingKeeping track. Stock, orders and holds carry over week to week in each 16‑week season, and a material the agent stops tracking can run dry.112–166598%
Coffee web‑shop order desk through a four‑act Christmas peakSmall businessChecking. Jobs the agent schedules keep running between its sessions, so a mistake in one goes unseen unless it checks its own records.207–213594%
Three‑day hospital ward care coordination, notes‑only memoryHospitalChecking. Results never arrive unasked, bookings get cancelled and orders change, so the agent has to keep checking and keep every task current.290–4156100%
Server‑room cooling outage in a 3D building with model‑played staffFacilitiesIn order. Temperatures climb every minute the cooling is down, so the order of the agent's moves decides whether the servers survive.31733%
Server‑hall cooling diagnosis by undoable tests in a 3D buildingFacilitiesKeeping track. Tests change covers, tiles, setpoints or the unit on duty, and the agent has to put each back as it found it.20750%
800 kg air handler move proved by trial in a 3D physics modelFacilitiesIn order. The unit's supports, loads and tilt carry from move to move, so the steps must come in an order that does no harm.18744%
Watchdog program commissioned for a running solvent recovery unitProcess controlChecking. Writes land on a running plant at once, so the agent can rehearse on a copy first, and must stay within its agreed scope.41693%
Room furnishing from photos: measure, match look‑alikes, order and planFurnishing serviceIn order. Orders charge at once and the delivery fee comes back only if a whole order is cancelled, so a hasty order costs money.28880%
Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacementIn time. Each order charges at once with its own delivery date, which has to fall inside a fixed window.25885%
Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove managementIn time. Each order charges at once and has to arrive on one of two days, within the family's budget.45892%
Night repair of a leaking condo floor on an in‑world clockHouseholdIn time. Each layer laid changes the floor's height, materials run out, and the clock runs to a 01:30 deadline.137100%
Club race‑car design and parts procurement on a web app with hidden supplier termsRace‑car engineeringKeeping track. Every installed part needs a matching order that arrives in time, and a redesign after ordering can void one.452.0% slower

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward. For the race car, Best is how much slower the closest car lapped than the fastest achievable car.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.