Tool use in stateful environments

The grader reads the world, not the agent's report

Interfaces

Command‑line tools that talk to a world service or simulator in 10 of the 19; a text browser with no shell in the two trip rebookings and the room furnishing; a live portal shown as numbered lines of text, one action a turn, in the two course registrations.

Other ways in

In the two server‑room jobs, one command, wa, acts in a 3D building; in two app builds, the agent's own app buys train seats on a practice site or runs a simulated house.

What changes

Bookings, payments and transfers; a bakery's orders and rota; truck plans; airline bookings, refunds and seats; registered classes; purchase orders; stock plans; ward tasks and pages; weekly offers and auction lots; furniture orders; tickets an app buys; a house's devices; server‑room cooling; an order desk's work orders and remedies.

What's graded

The final state and the action log. The course registrations read only what the portal holds when the agent stops: classes left in the cart and the closing message count for nothing. On the ward, the grader reads only the logged commands.

Mistakes let through

The registration portal blocks nothing: it registers clashes, full classes as waitlist places and classes without their prerequisites, flagging clashes, full classes and long walks but never a missing prerequisite. The ordering systems on the packing line, in spares buying and at the order desk refuse without saying why.

Orders checked by code

In the seasonal bulk order, an LLM reads which actions a message orders, from a fixed list for that person. Code then allows only what that person may do, with every figure and name taken from the message or records, and echoes back exactly what was done.

State that moves

Payments land after each household's own delay at the family dinner; seats sell out at fixed in‑world times as each action moves the cancelled‑flight rebooking's clock (the other's is frozen); heat builds each world minute in the server rooms; simulators move on between sessions.

One world per attempt

Each attempt gets its own world, logged action by action. In six simulators the grader replays the log through a fresh copy and checks that it matches the attempt's trace; the portal's offline grader rebuilds each final registration and reaches the same verdict.

Who decides

Code decides most of it. Judges or readers decide some, such as 5 of the dinner's 93 checks, note items worth 13 and 8 points in the two trip rebookings, 7 of 26 and 10 of 121 checks in two app builds, and 12 report checks in each server‑room job.

They trust what the system accepted

Each count covers every model tested on the environment, with only the request; where a model made several attempts, the line says so.

1. Silent acceptance trusted. Course registration against rules found in portal: 4 of 4 attempts, one per model, kept a starting class whose prerequisite the student hadn't passed, which the portal had registered without objection; none opened its page. Six registrations the catalogue allows meet every rule.

2. Seats lost while checking. Cancelled‑flight family rebooking: in 14 of 24 attempts (8 models × 3), the free seats for two travellers sold out while the agent was still asking or checking, and all 14 bought other tickets. GPT‑6 Sol secured them first in all 3 of its attempts.

3. A notice taken as exact. Month‑long parcel fleet dispatch: 1 of 2 attempts timed departures to a hub's delay notice; the parcels were ready later, and the day missed its mark by under a point. The other had measured from the history how notices compared with real ready times.

4. A report the stock contradicts. Two‑site spares pick lists: 2 of 6 attempts counted a special lamp as cover for the ordinary one, which no record supported. The final stock check failed at both sites, while both attempts reported their stock within the limit.

5. One site of two. Two‑site spares buying: 2 of 5 attempts first filed replacements only for the site whose records name parts in words, not the one using numbers, and 3 repairs waited. One found its gap two weeks later, then held both orders for next month's budget.

Full solves in seven of the nineteen, all but one multi‑session

Each dot is one attempt with only the request; the line marks the median. Larger dots are full solves.

Hover a dot for its model and reward.

What each one lets the agent change

Checks: a range where the count varies by attempt. Told the rules: in the family dinner it also gives what the people would say and two facts from the files, and in the room furnishing the measured floor plan and the list of pieces. A dash: not tested.

What we haven't measured

• Doing little can score: an empty registration meets 11 of 14 checks in both course registrations; a read‑only lookup scores about 20% in one flight rebooking, doing nothing about 30% in the other; idle agents earn 39% and 18% in the spares environments and about half at the order desk.
• Where the grader reads only the final state, a closing message that misreports it costs nothing: one course‑registration attempt said the day fit the parking permit when it didn't.
• Two of the 19 have results from only one or two models, and repeat attempts exist for only five, so how much attempts vary is mostly unmeasured.
• At the coffee roaster's materials planning, Claude Code's 4 attempts ran outside a container, beside the simulator and grader; transcripts show none read them. 6 of the 24 cancelled‑flight rebooking attempts with only the request met a confirmed fault in the airline's chat engine and still count.
• In the dinner, the bulk order, the two trip rebookings and the two server‑room jobs, an LLM voices the people or reads their messages; we haven't measured the reward under another model there.