Unattended train‑ticket buyer for a rate‑limited booking site

Fast enough to win returned seats, slow enough to stay inside the site's rules

Role

Sole builder on a commuter's laptop, with one account note and a practice booking site that has no API document.

Request

An app that keeps checking the site and buys each missing weekend journey with the saved card as seats appear: one seat per direction inside the Friday and Sunday time windows, never a journey already ticketed, and a control page to start and stop it.

Deliverable

A running worker and a control page showing its state, the journeys still missing and the details of each new ticket, plus a short note.

What the reward depends on

All eight missing journeys bought, a Sunday released at midnight among them, read from the site's own log; no hand‑bought ticket bought again; the site's pause and login limits kept; Start and Stop only the owner can use.

Where the requirements are

In the user's message, the account note and the site's own pages: the Terms page sets the search, login, session and hold rules and the release time. There is no API document.

A practice booking site with its own clock, holds and lockouts

Interface

A terminal in a fresh Docker container, driven by Claude Code, Codex CLI or, for Gemini 4 Argon, Antigravity CLI. Web search and fetch are off.

The site

A private practice booking site with a timetable, search, seat maps, seat holds that expire, saved‑card payment, bookings and Terms, its own clock, seeded stock and two tickets already bought by hand. It logs every request.

Other buyers

Seeded rules, not agents: released seats on the busy trains sell out within seconds, and returned seats come back briefly at seeded moments.

The site's limits

Published limits on searches, logins and session length; going over them pauses, suspends or locks the account.

Grading

A fresh copy of the site on a staged clock that crosses midnight, with a traffic capture. An operator model, GPT‑6.1 Sol, presses Start, then Stop, then Start again; a new Sunday goes on sale at midnight.

Limits

One build session of 120 minutes, 150 model responses and $20 at list prices.

26 checks in three weighted sections; tickets count only from the site's log

Structure

Build: 17 checks share 60 points, the eight missing journeys among them. Edge cases: 5 share 20, the site's rules and the midnight release. Security: 4 share 20.

Who decides

Scripts decide 19 checks from the site's log and scans of the app for secrets; two LLM judges, GPT‑6.1 Sol and GPT‑5.6 Luna, decide 7, and Opus 5.5 ruled their split votes. 43 of 624 verdicts are marked ruled, 15 of them scripted ones re‑decided after an audit.

What counts

Holds, rows on the app's control page or claims in the note never count as tickets; control times come from the real requests; Start and Stop are replayed from another network and from a cross‑site page.

Formula

Reward = (60 × build passed / 17 + 20 × edge passed / 5 + 20 × security passed / 4) / 100. An app that can't run fails all 26 checks.

One model bought all eight journeys; told the rules, four did

With only the request, Opus 5.5 met all 26 checks, two apps scored 96%, and two attempts left no working app. A hint lifted the mean from 65% to 76%, as those two recovered, but the median fell from 83% to 79%. Told the rules, four models fully solved it.

Each dot is one attempt; the line marks the median.

One attempt per model and version. Gemini 4 Argon was tested on this environment only.

With a hint: the request plus 6 points from someone familiar with the job.

Told the rules: the request plus 25 product requirements from an IT specialist.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request.

1. Journeys left unbought. 7 of 8 attempts missed at least one of the eight journeys, 34 of 64 in all; 5 of the 6 apps that ran checked each trip too seldom to catch a returned seat.

2. Too cautious. One app ran all session but held and bought no seat, pacing far below the site's limit.

3. No working app. 2 of 8 attempts left none: one wrote no app, the other couldn't find the account note.

4. Controls open to other sites. 1 of the 6 running apps took Start and Stop from any form post, so another website could control it.

What an attempt costs

Length

With only the request, the median attempt took 12 minutes and about 30 tool calls (19 to 67 across the 8).

Cost

The median attempt cost $1.65 at list prices, from $0.22 to $5.93 across the 8; all 24 attempts came to $59. GPT‑6 Luna has no published price, so its figures use the dearest default rate and are upper bounds; Gemini 4 Argon's are its logged tokens at Google's list price.

What's counted

The agent model only: the operator's and judges' calls aren't included.

What you license

With a licence

Revision 1 in all three versions: the account note, the practice site with its seeded stock and limits, the staged‑clock grading harness with the traffic capture, the judges' instructions, and the grader's report on each of the 24 attempts, as baseline results. No reference app is stored.

Format

The build runs in Docker with the site on the attempt's own network. Grading runs each app in real time, about 75 minutes and one app at a time, on a dedicated Linux VM whose clock is staged. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The site, grader and judges' instructions aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version.
• No reference app is stored: the self‑test's synthetic evidence scores 100, and an app that can't run scores 0.
• Grading runs in real time on a staged clock, and returned seats are free only briefly, so the reward depends on polling speed as well as care.
• The other buyers are seeded rules, not agents, so the timing of released and returned seats is the same in every attempt.