Transit journey planner and phone telemetry over five field weeks
ID tb‑tasks/farelane‑transit · version 1 · results as of 28 October 2026
As the engineer of a city's journey‑planning app, the agent builds a route planner and a program for riders' phones, fixes both after a week in the field and ships a second release after four, from what riders ask, do and say and what their phones record.
- Checks
- 81, 80 graded requests and a sandbox gate
- Tested
- 5 models, one attempt each
- Best
- 98% (GPT‑6 Astra)
- Median
- 84%
- Full solves
- 0 of 5
- Median attempt
- 115 min, 168 tool calls
A journey planner built, patched and re‑released over five weeks of riders
Long‑horizon work
Over three sessions in five weeks, later sessions see the phones' sensor data only as the agent's own code kept it.
Agentic coding
A planner and a sandboxed program run on every rider's request and trip, under CPU, memory and byte limits.
Clarifying with people
The planner can ask each rider at most four quick‑reply questions a request, so each one has to count.
Multimodal records
Some facts show only in phone sensor streams, which the agent's code must shrink to a small daily budget.
When to act or escalate
From a rider's fifth trip on a journey, extra questions annoy, and three annoyances lose the rider; too few leave needs unknown.
Catching errors
Nearly half the graded requests turn on places where the operator's timetable feed is wrong or silent.
A simulated city, 120 riders and their phones, over five weeks
The agent's world
A shell in a Linux container, one fresh session per release, with a client that checks the planner on pilot data and deploys it.
The field
Two metro lines, three tram lines, twelve bus routes, an express bus and a ferry, running as the city really runs for five weeks.
Riders
120 scripted riders with their own places, habits and comfort needs make one to three requests a day, and their phones record every trip.
The reward
The share of 80 week‑five requests planned right against an expert's plan for the rider's mode; a lost rider fails every later request.
Four of the five planned 81–98% of requests right, and the fifth 19%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 98% |
| GPT‑6.1 SolOpenAI | 96% |
| Opus 5.5Anthropic | 84% |
| Sonnet 5.5Anthropic | 81% |
| GPT‑6 LunaOpenAI | 19% |
GPT‑6 Astra and GPT‑6.1 Sol also passed the task's own bar; Opus 5.5 and Sonnet 5.5 missed it after losing riders in weeks 2 to 4.
They miss faults in week one's records, ask too much, and miss a crowded line
Long‑horizon work
60% lost riders in weeks 2 to 4 to faults already in week one's records: plans to the wrong place, needless questions, or a station whose lift was out.
Catching errors
Two never checked which plans riders had refused, the only sign of a plan to the wrong place.
When to act or escalate
Two miscounted repeat journeys and asked the full set of questions on trips riders made often: 12 and 15 needless questions, costing 3 and 5 riders.
Multimodal records
60% never found the metro line that's standing‑only one way in the morning peak, which cost Opus 5.5 two graded requests.
Agentic coding
One read a bug in its own question messages as questions being refused and never asked any; it lost 81 of 120 riders and planned 15 of 80 right.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.06 to $32.99 for the agent model; grading is a script, with no model calls.
What's in it
The simulator and riders, documents and pilot set, client and sandbox, grader and expert plans, and a reference app that sees the field's hidden truth.
Format
A Harbor task in a Linux container, with the field simulator running as root beside the agent.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials