AsapiThe AI Agent Trials
Asapi / Catalogue / Transit journey planner and phone telemetry over five field weeks
Environment datasheet · Software for a transit operator

Transit journey planner and phone telemetry over five field weeks

ID tb‑tasks/farelane‑transit · version 1 · results as of 28 October 2026

As the engineer of a city's journey‑planning app, the agent builds a route planner and a program for riders' phones, fixes both after a week in the field and ships a second release after four, from what riders ask, do and say and what their phones record.

Checks
81, 80 graded requests and a sandbox gate
Tested
5 models, one attempt each
Best
98% (GPT‑6 Astra)
Median
84%
Full solves
0 of 5
Median attempt
115 min, 168 tool calls
What it tests

A journey planner built, patched and re‑released over five weeks of riders

Long‑horizon work

Over three sessions in five weeks, later sessions see the phones' sensor data only as the agent's own code kept it.

Agentic coding

A planner and a sandboxed program run on every rider's request and trip, under CPU, memory and byte limits.

Clarifying with people

The planner can ask each rider at most four quick‑reply questions a request, so each one has to count.

Multimodal records

Some facts show only in phone sensor streams, which the agent's code must shrink to a small daily budget.

When to act or escalate

From a rider's fifth trip on a journey, extra questions annoy, and three annoyances lose the rider; too few leave needs unknown.

Catching errors

Nearly half the graded requests turn on places where the operator's timetable feed is wrong or silent.

How it works

A simulated city, 120 riders and their phones, over five weeks

The agent's world

A shell in a Linux container, one fresh session per release, with a client that checks the planner on pilot data and deploys it.

The field

Two metro lines, three tram lines, twelve bus routes, an express bus and a ferry, running as the city really runs for five weeks.

Riders

120 scripted riders with their own places, habits and comfort needs make one to three requests a day, and their phones record every trip.

The reward

The share of 80 week‑five requests planned right against an expert's plan for the rider's mode; a lost rider fails every later request.

Results

Four of the five planned 81–98% of requests right, and the fifth 19%

Each dot is one attempt, and the line marks the median.

MedianFully solved
5 modelsOne attempt each84%0 of 5
ModelReward
GPT‑6 AstraOpenAI98%
GPT‑6.1 SolOpenAI96%
Opus 5.5Anthropic84%
Sonnet 5.5Anthropic81%
GPT‑6 LunaOpenAI19%

GPT‑6 Astra and GPT‑6.1 Sol also passed the task's own bar; Opus 5.5 and Sonnet 5.5 missed it after losing riders in weeks 2 to 4.

What models get wrong

They miss faults in week one's records, ask too much, and miss a crowded line

Long‑horizon work

  1. 60% lost riders in weeks 2 to 4 to faults already in week one's records: plans to the wrong place, needless questions, or a station whose lift was out.

Catching errors

  1. Two never checked which plans riders had refused, the only sign of a plan to the wrong place.

When to act or escalate

  1. Two miscounted repeat journeys and asked the full set of questions on trips riders made often: 12 and 15 needless questions, costing 3 and 5 riders.

Multimodal records

  1. 60% never found the metro line that's standing‑only one way in the morning peak, which cost Opus 5.5 two graded requests.

Agentic coding

  1. One read a bug in its own question messages as questions being refused and never asked any; it lost 81 of 120 riders and planned 15 of 80 right.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.06 to $32.99 for the agent model; grading is a script, with no model calls.

What's in it

The simulator and riders, documents and pilot set, client and sandbox, grader and expert plans, and a reference app that sees the field's hidden truth.

Format

A Harbor task in a Linux container, with the field simulator running as root beside the agent.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.