Course registration against rules found in portal
ID northgate/task1‑1 · revision 2 · results as of 9 November 2026
On a university's registration portal, read as text one page at a time, the agent fixes a student's term timetable so that it fits commitments, scholarship terms and degree rules that only the student's records state.
- Checks
- 14, 11 of them met by an empty registration
- Tested
- 16 models, one attempt each
- Best
- 100% (GPT‑6 Astra and GPT‑6.1 Sol)
- Median
- 71%
- Full solves
- 2 of 16
- Median attempt
- 6 min, 34 tool calls
A term's timetable fixed around rules only the student's records state
Stateful tool use
The portal registers whatever it's given, so the agent has to check the final timetable against rules set out on its own pages.
Catching errors
The starting registration holds problem sections the portal accepts silently, such as a class before the car park opens.
Unstated requirements
The request only says commitments exist; work shifts, training, the car park's hours and scholarship terms sit in portal pages.
A live portal read as text, one page and one action at a time
The agent's world
A harness shows each portal page as numbered lines of text; the model answers with one action: click, type, select, go, back or stop.
The portal
87 pages, including 51 course pages with 108 sections; it blocks nothing, registering clashes, full sections and missing prerequisites.
Limits
Up to 250 replies, with a fixed in‑world date and nobody to answer; loops of repeated actions end an attempt.
The reward
14 checks by script on the registration the portal holds; an empty one meets 11, since those checks only look for broken rules.
56% of attempts scored below the 79% an empty registration earns; two met all 14 checks
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 100% |
| GPT‑6.1 SolOpenAI | 100% |
| GPT‑6.1 TerraOpenAI | 86% |
| Fable 5.1Anthropic | 79% |
| GPT‑6 SolOpenAI | 79% |
| Opus 5Anthropic | 79% |
| Opus 5.5Anthropic | 79% |
| GPT‑5.6 SolOpenAI | 71% |
| Sonnet 5.5Anthropic | 71% |
| GPT‑5.6 TerraOpenAI | 64% |
| GPT‑6 LunaOpenAI | 64% |
| Opus 4.8Anthropic | 64% |
| Haiku 4.5Anthropic | 57% |
| Sonnet 5Anthropic | 57% |
| GPT‑5.6 LunaOpenAI | 50% |
| Haiku 5.5Anthropic | 43% |
They trust what the portal accepts, miss the advisor's course, and read commitments loosely
Catching errors
69% kept a starting course whose prerequisite the student hadn't passed, which the portal had registered without objection.
Unstated requirements
63% didn't register the course the advisor's note describes by content, and 75% closed too few degree requirements.
50% kept the starting class that begins before the car park opens, and 44% had a day on campus longer than the permit covers.
Stateful tool use
44% left a transfer too short to walk.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 640,000 tokens of input and output.
What's in it
The portal and its pages, the agent harness, the grader and its self‑test, and an answer key with a scripted reference.
Format
Packaged as a Harbor task on request; the portal and grader are standard‑library Python, and the harness drives the site in Chromium.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials