Asapi
The AI Agent Trials / Your AI race team
Benchmark · Motor racing

Your AI race team

Three jobs at a club race team. In one, the AI stands in for the team's engineer, designing the fastest car the rules allow on the team's website and buying every part from suppliers who give prices only when asked. In the other two, four AIs each drive a car round a simulated track from their own cockpit view, steering a moving machine from what a camera sees; in the second, crashes do lasting damage.

3 jobs · 7 AIs tested. GPT‑6.1 Sol, GPT‑6.1 Terra and GPT‑6 Astra finished both races, and GPT‑6 Astra's race car came closest to the fastest achievable car.

?!
The jobs

The three jobs

Every AI on every job, each a dot: a score from 0 to 100, or for the race car, how much slower its car was than the fastest achievable car. Open a job to see what the best AI got right and wrong.

The best resultAnother AIEverything right

Design and buy a race car (5 AIs tested)

A club race team's engineer has left before race week, so on the team's website the AI designs the fastest car the rules allow, then buys every part before race day from suppliers who give prices only when asked.

What the best AI did

  • Bought every part on time and within budget, for a car with none of the part combinations that had broken down in past races.
  • Settled on a car that would qualify 2.0% slower than the fastest achievable car.

How much slower each car was

GPT‑6 Astra2.0%
Opus 5.54.4%
GPT‑6 Sol4.6%
Opus 54.7%
GPT‑6 Luna6.8%
The AIs

Who did best

GPT‑6.1 Sol, GPT‑6.1 Terra and GPT‑6 Astra finished both races, Sol quickest each time and Astra never put back on the track. Only GPT‑6 Astra and GPT‑6 Luna were tested on all three jobs. Astra's race car was 2.0% slower than the fastest achievable car; Opus 5.5's, GPT‑6 Sol's (an older model than GPT‑6.1 Sol) and Opus 5's 4.4% to 4.7%. GPT‑6 Luna finished the first race last and retired from the second.

What we learned

What the three jobs showed

What they do well

They get a car round the track from the driver's seat, slowly.

All four AIs finished the first race, steering from cockpit pictures several seconds old by the time their steering reached the car. The slowest, GPT‑6 Luna, took more than half an hour, had to be put back on the track 13 times and was given driving tips.

They check the past results and buy carefully.

On the race car, all five AIs read the series' past results before choosing a car, and every car they submitted avoided the part combinations that had broken down. All five ordered every part to arrive by race day within the budget.

Where they fall short

They settle for a slower car than they could have bought.

The fastest design could never be bought, because a tyre it needed couldn't arrive in time. GPT‑6 Astra switched to another car and ended up 2.0% slower than the fastest achievable car. The other four ended on the same model of car, built differently, 4.4% to 6.8% slower, and two of them never chose any other.

Mistakes on the track add up.

GPT‑6 Luna got stuck and had to be reset 13 times in the first race. In the second, where crashes weaken the car for good, it crashed 21 times until its engine failed, and GPT‑6.1 Terra finished after 12 crashes and 13 times put back on the track.

For the curious

How these jobs were set up and scored

The garage
A club racing team's website, with a lap simulator, six suppliers' order pages and the racing series' past results. The AI can simulate as often as it likes, but must settle on a design before any supplier will talk to it, and it submits its car once.
The races
Each AI sees only its own cockpit view and dashboard, and the race keeps moving while it thinks. A car stuck for eight seconds can ask to be reset, at a cost of 20 seconds. In the second race, crashes permanently weaken the engine, brakes, grip and steering, and a badly damaged car is out.
Scores
A race scores 100 for finishing a lap and 0 for retiring; speed doesn't count. On the race car job, a car is out if it breaks a rule, repeats a breakdown from past results, or isn't fully bought on time within budget. Every other car is compared with the fastest achievable car, the fastest that clears all three, by how much slower its qualifying lap was after the handicap the racing series gives each kind of car.
How the races were run
We scored one run of each race, with all four AIs on the track together. GPT‑6 Astra also coordinated both races and came in with prior context; the other three started fresh. In the first race, GPT‑6 Luna was given general driving tips once the others had finished.
Time and cost
A race took each car 6 to 33 minutes of real time. Designing and buying the race car took 11 to 62 minutes. Cost wasn't recorded for most of them.

Every AI's result on every job

AIDesign and buy a race carDrive a lap from the driver's seatDrive a lap as crash damage builds up
GPT‑6 AstraOpenAI2.0% slower100100
GPT‑6 LunaOpenAI6.8% slower1000
GPT‑6.1 SolOpenAI–100100
GPT‑6.1 TerraOpenAI–100100
GPT‑6 SolOpenAI4.6% slower––
Opus 5Anthropic4.7% slower––
Opus 5.5Anthropic4.4% slower––

Each AI was tested once. The race car shows how much slower each car was than the fastest achievable car; a race scores 100 for a finished lap and 0 for retiring. A dash means we didn't test that AI on that job.