Unattended train‑ticket buyer for a rate‑limited booking site
ID laycode/task09 · revision 1 · results as of 28 October 2026
A weekly commuter asks for an app that buys one seat each way for every open weekend still missing a ticket, inside their travel windows and the booking site's limits, with Start, Stop and a page showing what it bought. The agent builds it from one account note.
- Checks
- 26, in three weighted parts
- Tested
- 8 models, one attempt each
- Best
- 100% (Opus 5.5)
- Median
- 83%
- Full solves
- 1 of 8
- Median attempt
- 12 min, 30 tool calls
An app that buys missing train seats on its own, inside the site's limits
Stateful tool use
The app watches a site with its own clock, where returned seats come back briefly and seat holds expire.
When to act or escalate
Returned seats last under two minutes, so checking too seldom loses them, while going over the site's limits pauses the account.
Unstated requirements
Seven checks rest on facts outside the request, such as the pause and login limits on the site's Terms page.
Agentic coding
The buyer runs unattended against a copy of the site, and a ticket counts only if the site's own log shows it.
A practice booking site, a staged clock, and seats released at midnight
The agent's world
A fresh container with one account note and a practice booking site with no API document, worked through the model's usual coding agent.
The site
Seat maps, expiring holds, card payment, published limits and its own clock; returned seats appear at set moments, and the commuter already holds two tickets.
Grading
On a fresh site copy with a staged clock past midnight, an operator model presses Start, Stop and Start; two AI judges decide 7 checks.
The reward
26 checks: 17 features share 60%, the eight journeys among them, then edge cases 20% and security 20%; only tickets in the site's log count.
Five attempts scored 79–100%, one 57%, and two left no working app
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 100% |
| Gemini 4 ArgonGoogle | 96% |
| Sonnet 5.5Anthropic | 96% |
| Fable 5.1Anthropic | 88% |
| GPT‑6 AstraOpenAI | 79% |
| GPT‑6.1 SolOpenAI | 57% |
| GPT‑6 LunaOpenAI | 0% |
| Sonnet 5Anthropic | 0% |
With an IT specialist's 25 requirements added to the request, four of the eight met all 26 checks.
They check too seldom, play it too safe, or leave no working app
When to act or escalate
88% of attempts missed at least one of the eight journeys, 34 of 64 in all; five of the six running apps checked too seldom to catch a returned seat.
One app ran all session but held and bought no seat, pacing itself far below the site's limit.
Agentic coding
Two attempts left no working app: one wrote none, and the other couldn't find the account note.
One of the six running apps took Start and Stop from any form post, so another website could control it.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $1.65 at list prices for the agent model; the operator's and judges' calls are extra.
What's in it
The account note, the practice site with its stock and limits, the staged‑clock grader, the task in three versions, and per‑check results.
Format
Packaged as a Harbor task on request; grading runs each app for about 75 minutes on a Linux machine whose clock crosses midnight.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials