Club race‑car design and parts procurement on a web app with hidden supplier terms
ID swap‑garage/task01 · revision 1 · results as of 17 November 2026
On a club racing team's garage website, the agent picks one of 19 chassis, builds and simulates the fastest car the class allows, and freezes the design. Then it buys every part before race day and within budget, from six suppliers who give stock, delivery and price only when asked.
- Checks
- 4, pass‑fail gates: a legal car, no part combination that has broken down, a design frozen first, and every part ordered in time, within budget and to the car's specification
- Tested
- 5 models, one attempt each
- Best
- 2.0% slower than the fastest achievable car (GPT‑6 Astra)
- Median
- 4.6% slower
- Full solves
- 0 of 5
- Median attempt
- 19 min, 575 tool calls
Design the fastest car the rules allow, then buy it from suppliers who quote only when asked
Engineering design
The reward compares the car's handicapped lap with the fastest achievable car's, the fastest buyable, breakdown‑free build among 42.7 million legal builds of 19 chassis.
Unstated requirements
Six fast pairs of parts have broken down in past races; only the series' past results show it, and a car using one scores 0.
Stateful tool use
The design must be frozen before suppliers answer, every installed part needs a matching on‑time order, and a redesign can void one.
Clarifying with people
Six scripted suppliers give stock, delivery and price only when asked, the only way to find the part that can't arrive in time.
A garage website, six supplier portals, four seasons of results and one submission
The agent's world
A headless browser as its only tool, on a sealed network with no shell, files or web, for 90 minutes.
The garage
19 chassis with published handicaps, a lap simulator to run at will, and one design frozen before suppliers answer; redesigns may follow.
The suppliers
Six scripted workshops answer stock, price and delivery questions, with an $11,400 budget and race day on day 7.
The reward
A car that fails any of four gates earns a reward of 0; otherwise, the fastest achievable car's handicapped lap divided by its own.
Every car passed all four gates, and the closest lapped 2.0% slower than the fastest achievable car
Each dot is one attempt, placed by how much slower its car lapped than the fastest achievable car, and the line marks the median.
| Model | Slower than the fastest achievable car |
|---|---|
| GPT‑6 AstraOpenAI | 2.0% |
| Opus 5.5Anthropic | 4.4% |
| GPT‑6 SolOpenAI | 4.6% |
| Opus 5Anthropic | 4.7% |
| GPT‑6 LunaOpenAI | 6.8% |
As rewards, these attempts earned 94% to 98%: the fastest achievable car's handicapped lap divided by each car's own. Only the fastest achievable car itself would be a full solve; a stock car of one chassis, bought in full, earns about 78%, 29% slower.
They settle for a slower car than they could have bought, and four end on the same chassis
Engineering design
The submitted cars lapped 2.0% to 6.8% slower than the fastest achievable car; 673 to 79,651 buyable builds were faster.
Four of five submitted the same chassis and lapped 4.4% to 6.8% slower, and two never chose any other; GPT‑6 Astra switched chassis after its first design couldn't be bought, and lapped 2.0% slower.
When its tyre couldn't arrive by race day, Opus 5.5 changed tyres and tested other parts but kept the chassis, though the fastest achievable car was on another.
What an attempt takes, and what the environment includes
An attempt
Attempts took 11 to 62 minutes, with 16 to 47 simulations and 437 to 836 tool calls.
What's in it
The garage, supplier and results site, the lap simulator, and the grading server with its exhaustive table of legal builds.
Format
Three Docker images, the site's server, an egress proxy and the agent's browser, on one sealed network; other packaging on request.
Fresh variants
Other seeded episodes in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows how much slower each model's car was.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
Asapi