Sequential decisions under time and uncertainty

A plan is scored by what it would do

Cost models

Two equipment plans are priced by a profit model over 12 or 36 months, from the manager's own plan or patching the old unit (0) to the best plan found (1): by exhaustive search of 51,121 candidates, 18,688 of them valid, and in a search of 41,268 valid plans.

Margins

In the 12‑month plan, a valid plan within a fixed margin of the best scores 1. The three‑year plan has no band, and its best plan stays best over any horizon from 15 to 36 months.

Profit with checks

In the seasonal bulk order, 70% of the reward is profit played out from the final state: from saying yes to the whole order as things stand (0) to the most profitable outcome the staff can carry out (1).

Household plans

The solar plan is scored on paying routes over time against a rebate deadline and a card limit. The savings plan's fund and target months are checked against bands a cash‑flow model computes from the plan's own figures. The smart‑home plan must fit a budget and wire up in simulation.

Commitments in simulators

Daily delivery windows with a pass mark; doses and tests on a ward; items on site before their stage; stock plans checked at weeks 4 and 8; a bank limit that moves with unpaid invoices; coffee bought ahead of seasons and disruptions; an order desk's stock through a Christmas peak.

A clock that moves

In the flight rebooking, each page, form and message moves an in‑world clock, and seats on alternative flights sell out at fixed times, the same in every attempt. In the server‑room incident, a phone call costs five world minutes while the room heats.

Uncertainty

Some suppliers' real lead times differ from their quotes; one sales tip becomes an order and another doesn't; a forecast is too high and a closure warning is withdrawn; trucks wear out; buyers pay late. The facts sit in records and with people.

Optimality

In the course registration for earliest graduation, one check passes only for the class the answer key takes as the route to graduating soonest, of three sharing a time slot; keeping the class already held breaks no rule.

Who decides

Code alone in 12 of the 18. In the bulk order, the solar plan, the savings plan, the cancelled‑flight rebooking, the smart‑home build and the server‑room incident, two LLM judges also decide checks on the plan, note, report or app (in the smart‑home build, possibly one Codex session).

They take the plan they're handed, and don't look ahead

Each count covers every model tested on the environment, with only the request; where a model made several attempts, the line says so.

1. The request taken as the plan. 12‑month equipment plan: 8 of 13 attempts, one per model, ordered only the cabinet the manager asked for and scored 0; a ninth added more and scored 33%. The four that found the old one's fault and repaired it scored 44–100%.

2. The listing trusted. Three‑year repair‑or‑replace plan: 6 of 12 attempts never asked when a new cabinet could arrive, missed the stock delay behind the listed time, ordered no stop‑gap repair and left the shop with no ice cream to sell over Easter. The other six patched the old one.

3. Asking too late. Tea‑factory cash desk: in the hardest scenario that needs the owner's money, where the largest credit buyer stops paying, 2 of 5 attempts asked too late, and the factory went under before it arrived. Opus 5.5 and Sonnet 5.5 asked in time from the same books.

4. The shops left short. Seasonal bulk order: 8 of 8 attempts, one per model, left the bakery's own shops short in at least one week, against the owner's standing rule that the shops come first. Told the rules, 5 of 8 kept them whole.

5. The real tip left uncovered. Coffee roaster materials planning: in 9 of the 13 windows that reached it, no coffee was bought or put on hold for the sales tip that became an order; GPT‑6.1 Sol held it in both its windows.

6. Calls before the stopgap. Server‑room cooling incident: in 7 of 7 attempts, one per model, the agent made calls or chased the cause first, two never starting a stopgap, and the servers shut down on heat 15 to 20 minutes in.

20 full solves, in seven of the eighteen

Each dot is one attempt with only the request; the line marks the median. Larger dots are full solves.

Hover a dot for its model and reward.

How each one prices the plan

Cost model: the reward prices the agent's plan, from a baseline plan (0) to the best a search found (1). Told the rules: both equipment plans get the manager's records, and the room furnishing the measured floor plan and piece list. A range: varies by attempt. A dash: not tested.

What we haven't measured

• 24 attempts on the 12‑month equipment plan, 7 of them with only the request, were scored against an earlier wording of the manager's records, with that wording's own best plan.
• The optimality check in the course registration for earliest graduation is disputed: an audit found nothing on the portal showing the rewarded class gets the student out sooner. The owner kept it, as the student's stated wish decides.
• Doing little scores in some: an idle agent earns 39% and 18% in the two spares environments and an empty course registration 79%; in the bulk order, an audit's near‑empty test copy scored about 55%; doing nothing earns about half an act at the order desk.
• At the coffee roaster's materials planning, Claude Code's 4 attempts ran outside a container, beside the simulator and grader; transcripts were checked afterwards. One restraint check there counts all coffee bought for a set period after it, even for a firm order or under a hold the policy allows.
• Two of the 18 have results from only one or two models, and repeat attempts exist in four, so how much attempts vary is mostly unmeasured.