AsapiThe AI Agent Trials
Asapi / Catalogue / Unattended train‑ticket buyer for a rate‑limited booking site
Environment datasheet · Household

Unattended train‑ticket buyer for a rate‑limited booking site

ID laycode/task09 · revision 1 · results as of 28 October 2026

A weekly commuter asks for an app that buys one seat each way for every open weekend still missing a ticket, inside their travel windows and the booking site's limits, with Start, Stop and a page showing what it bought. The agent builds it from one account note.

Checks
26, in three weighted parts
Tested
8 models, one attempt each
Best
100% (Opus 5.5)
Median
83%
Full solves
1 of 8
Median attempt
12 min, 30 tool calls
What it tests

An app that buys missing train seats on its own, inside the site's limits

Stateful tool use

The app watches a site with its own clock, where returned seats come back briefly and seat holds expire.

When to act or escalate

Returned seats last under two minutes, so checking too seldom loses them, while going over the site's limits pauses the account.

Unstated requirements

Seven checks rest on facts outside the request, such as the pause and login limits on the site's Terms page.

Agentic coding

The buyer runs unattended against a copy of the site, and a ticket counts only if the site's own log shows it.

How it works

A practice booking site, a staged clock, and seats released at midnight

The agent's world

A fresh container with one account note and a practice booking site with no API document, worked through the model's usual coding agent.

The site

Seat maps, expiring holds, card payment, published limits and its own clock; returned seats appear at set moments, and the commuter already holds two tickets.

Grading

On a fresh site copy with a staged clock past midnight, an operator model presses Start, Stop and Start; two AI judges decide 7 checks.

The reward

26 checks: 17 features share 60%, the eight journeys among them, then edge cases 20% and security 20%; only tickets in the site's log count.

Results

Five attempts scored 79–100%, one 57%, and two left no working app

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 modelsOne attempt each83%1 of 8
ModelReward
Opus 5.5Anthropic100%
Gemini 4 ArgonGoogle96%
Sonnet 5.5Anthropic96%
Fable 5.1Anthropic88%
GPT‑6 AstraOpenAI79%
GPT‑6.1 SolOpenAI57%
GPT‑6 LunaOpenAI0%
Sonnet 5Anthropic0%

With an IT specialist's 25 requirements added to the request, four of the eight met all 26 checks.

What models get wrong

They check too seldom, play it too safe, or leave no working app

When to act or escalate

  1. 88% of attempts missed at least one of the eight journeys, 34 of 64 in all; five of the six running apps checked too seldom to catch a returned seat.

  2. One app ran all session but held and bought no seat, pacing itself far below the site's limit.

Agentic coding

  1. Two attempts left no working app: one wrote none, and the other couldn't find the account note.

  2. One of the six running apps took Start and Stop from any form post, so another website could control it.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $1.65 at list prices for the agent model; the operator's and judges' calls are extra.

What's in it

The account note, the practice site with its stock and limits, the staged‑clock grader, the task in three versions, and per‑check results.

Format

Packaged as a Harbor task on request; grading runs each app for about 75 minutes on a Linux machine whose clock crosses midnight.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.