Transit journey planner and phone telemetry over five field weeks

The timetable is wrong in a dozen ways that only the field shows

Role

The engineer of a transit operator's journey‑planning app, over three sessions: a build on day 0, a patch after one week and a second release after four.

Request

Write the planner, which reads each rider's own‑words request and chosen mode, may ask up to four quick‑reply questions and answers with a plan, and the hook that keeps what it chooses of each trip's sensor streams within a byte budget.

Deliverable

A planner and a hook, deployed with the task's client in each session; the last deploy of the second release is graded.

What the reward depends on

Plans that work in the city as it really runs, matching an expert's for the mode each rider wanted, on 80 requests in week five that aren't identified to the agent; riders kept, since one who stops using the app fails every later request.

Where the requirements are

In the product brief, the operator's static timetable of about 114,000 stop times, the city's maps and a labelled pilot set of recorded trips and example requests, then in what riders ask, do and say, and what their phones record between sessions.

A simulated city, 120 scripted riders and their phones

Interface

A shell in a Linux container as an unprivileged user, one fresh session per release step, driven by Claude Code or Codex CLI, with a client that checks a planner and hook on pilot data and deploys them to a root‑owned field daemon.

Network

The container could reach the internet, though nothing the task needs is online. Codex's web search was off; Claude Code's web tools were on for Opus 5.5 and Sonnet 5.5, and neither used them.

The field

The city as it really runs, with two metro lines, three tram lines, twelve bus routes, an express bus and a ferry, which runs whatever was deployed through five simulated weeks.

Simulated riders

120, scripted with no live model: each with places, a schedule, a mode habit and comfort needs, making one to three requests a day in their own words. Their phrasings and feedback lines were written once by a model and frozen.

Phones

Every trip's raw accelerometer, gyroscope, magnetometer, barometer, GPS and screen streams reach only the agent's hook, which runs sandboxed with no network, processes or file writes.

Memory

Only the workspace persists between sessions; the harness's memory is cleared.

Limits

Agent time of 180, 60 and 180 minutes in the three sessions; attempts used 9 to 127 minutes in all.

80 graded requests against an expert's plans, script only

Structure

80 week‑five requests, chosen when the task was made and not identified to the agent: 38 that turn on where the field differs from the timetable, 10 on place names and luggage, and 32 with nothing hidden; plus a zero‑point gate that the hook stayed in its sandbox.

Who decides

The task's grader judges each request from the field's trace against the hidden expert's plan for the mode the rider wanted, on that day as it ran; script only. An audit fix on 28 October was applied by regrading every attempt.

Passing

A plan must first be usable in the field that day, then close enough to the expert's on the rider's mode: expected time (fastest), late‑end arrival (most reliable), the same transfers within the time tolerance (fewest transfers), or a comfort test with two readings.

Formula

Reward here = graded requests passed / 80; the package's reward.json keeps this as its score and gives a pass/fail reward, 1 only when a high share of the 80 pass and the hook stayed in its sandbox. Deploying nothing scores 0 on both.

Two models passed the task's own pass rule; none planned every request right

GPT‑6 Astra planned 78 of the 80 requests right and GPT‑6.1 Sol 77, and both passed the task. Opus 5.5 and Sonnet 5.5 planned 67 and 65 right but didn't pass, after losing riders, and GPT‑6 Luna planned 15.

Each dot is one attempt; the line marks the median.

One kept attempt per model, five models. GPT‑6.1 Sol's first attempt lost two sessions to provider capacity errors and was made again in full. Two attempts ran the day before version 1 was locked, on an earlier build with identical agent‑facing files, and were scored by a full regrade.

A smaller variant with 40 riders and 40 graded requests exists, used for development only.

What the models got wrong

Counted over the 5 models, one attempt each.

1. Riders lost to visible faults. 3 of 5 attempts lost riders in weeks 2 to 4 to faults already in week one's records: plans to the wrong place, needless questions, or a station whose lift was out. It cost Opus 5.5 and Sonnet 5.5 the task's pass.

2. Refused plans never checked. 2 of 5 never checked which plans riders had refused, the only sign of a plan to the wrong place.

3. Repeat journeys miscounted. 2 of 5 miscounted repeat journeys and asked the full set of questions on journeys riders made often: 12 and 15 needless questions in all, which cost 3 and 5 riders.

4. A standing‑only line missed. 3 of 5 never found the metro line that's standing‑only one way in the morning peak; only Opus 5.5's comfort plans kept riders standing on it.

5. A bug taken for a rule. 1 of 5 read a bug in its question messages as questions being refused and never asked anything, while its place matcher sent about one plan in nine to the wrong place. It lost 81 of 120 riders and planned 15 of 80 right.

What an attempt costs

Length

The median attempt took 115 minutes of agent time across its three sessions and 168 tool calls (57 to 377 across the 5).

Cost

Every figure is an estimate from the tokens used, since the attempts ran on plan sign‑ins and nothing was charged: a median of $13.77, from $0.06 to $32.99 across the 5, $74 in all. GPT‑6 Luna's rate is derived from GPT‑6 Sol's; GPT‑6 Astra's is the rate Harbor assumed.

What you license

With a licence

Version 1: the field simulator and its riders, the documents and labelled pilot set, the client and sandbox, the grader with the hidden expert plans, a reference app that passes given the field's hidden truth, and the grader's report on each of the 5 attempts, as baseline results.

Format

A Harbor task in a Linux container, with the field daemon running as root beside the agent.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The field, expert plans and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• The reference solution is given the field's hidden truth, so it shows the task can be passed, not that the facts can be learned; it plans 77 of the 80 right. An app built from the documents alone plans 40%.
• The riders are scripted, not live models, so their words and feedback repeat across attempts.
• Two attempts ran the day before version 1 was locked, on an earlier build whose agent‑facing files are identical, and were scored by a full regrade of their records.