Course registration optimised for earliest graduation
ID northgate/task1‑2 · revision 2 · results as of 9 November 2026
The same portal, student and commitments as the course‑registration environment, with one more wish: the student wants to graduate as soon as possible, and 22 sections are full. The agent has to weigh what each course opens up for later terms.
- Checks
- 14, 13 rules and 1 optimality check
- Tested
- 15 models, one attempt each
- Best
- 93% (GPT‑5.6 Sol and GPT‑6 Astra)
- Median
- 79%
- Full solves
- 0 of 15
- Median attempt
- 10 min, 45 tool calls
The same timetable, now planned for graduating as soon as possible
Stateful tool use
The portal registers whatever it's given, so the final timetable has to keep every rule and graduate the student soonest.
Planning ahead
One check rewards the same‑slot course that leads to graduating soonest, though keeping the course already held breaks no rule.
Catching errors
The starting registration holds problems, such as a course without its prerequisite that also meets on the student's work days.
Unstated requirements
Commitments, parking limits, scholarship terms and degree rules must be found across the portal's pages; the request names none.
A live portal read as text, one page and one action at a time
The agent's world
A harness shows each portal page as numbered lines of text; the model answers with one action: click, type, select, go, back or stop.
The portal
87 pages, including 51 course pages with 108 sections, 22 of them full; it blocks nothing, registering clashes and missing prerequisites.
Limits
Up to 250 replies, with a fixed in‑world date and nobody to answer; loops of repeated actions end an attempt.
The reward
13 rule checks and one optimality check by script, on the registration the portal holds; an empty one meets 11.
The best two scored 93%, each missing only the optimality check
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑5.6 SolOpenAI | 93% |
| GPT‑6 AstraOpenAI | 93% |
| GPT‑6 SolOpenAI | 86% |
| GPT‑6.1 SolOpenAI | 86% |
| GPT‑6.1 TerraOpenAI | 86% |
| Opus 5.5Anthropic | 86% |
| Fable 5.1Anthropic | 79% |
| Opus 5Anthropic | 79% |
| Sonnet 5.5Anthropic | 71% |
| GPT‑6 LunaOpenAI | 64% |
| GPT‑5.6 LunaOpenAI | 57% |
| GPT‑5.6 TerraOpenAI | 57% |
| Haiku 4.5Anthropic | 57% |
| Haiku 5.5Anthropic | 50% |
| Sonnet 5Anthropic | 50% |
They keep the course already held, trust what the portal accepts, and read commitments loosely
Planning ahead
93% kept the course already held in the contested slot, which breaks no rule, rather than the one leading to graduation soonest.
Catching errors
67% held a course whose prerequisite the student hadn't passed, which the portal registered without objection.
Unstated requirements
40% kept the starting class that begins before the car park opens, and 47% had a day on campus longer than the permit covers.
Stateful tool use
33% left a transfer too short to walk.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 1.4 million tokens of input and output.
What's in it
The portal and its pages, the agent harness, the grader and its self‑test, and an answer key with an optimal registration.
Format
Packaged as a Harbor task on request; the portal and grader are standard‑library Python, and the harness drives the site in Chromium.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials