AsapiThe AI Agent Trials
Asapi / Catalogue / Course registration against rules found in portal
Environment datasheet · University portal

Course registration against rules found in portal

ID northgate/task1‑1 · revision 2 · results as of 9 November 2026

On a university's registration portal, read as text one page at a time, the agent fixes a student's term timetable so that it fits commitments, scholarship terms and degree rules that only the student's records state.

Checks
14, 11 of them met by an empty registration
Tested
16 models, one attempt each
Best
100% (GPT‑6 Astra and GPT‑6.1 Sol)
Median
71%
Full solves
2 of 16
Median attempt
6 min, 34 tool calls
What it tests

A term's timetable fixed around rules only the student's records state

Stateful tool use

The portal registers whatever it's given, so the agent has to check the final timetable against rules set out on its own pages.

Catching errors

The starting registration holds problem sections the portal accepts silently, such as a class before the car park opens.

Unstated requirements

The request only says commitments exist; work shifts, training, the car park's hours and scholarship terms sit in portal pages.

How it works

A live portal read as text, one page and one action at a time

The agent's world

A harness shows each portal page as numbered lines of text; the model answers with one action: click, type, select, go, back or stop.

The portal

87 pages, including 51 course pages with 108 sections; it blocks nothing, registering clashes, full sections and missing prerequisites.

Limits

Up to 250 replies, with a fixed in‑world date and nobody to answer; loops of repeated actions end an attempt.

The reward

14 checks by script on the registration the portal holds; an empty one meets 11, since those checks only look for broken rules.

Results

56% of attempts scored below the 79% an empty registration earns; two met all 14 checks

Each dot is one attempt, and the line marks the median.

MedianFully solved
16 modelsOne attempt each71%2 of 16
ModelReward
GPT‑6 AstraOpenAI100%
GPT‑6.1 SolOpenAI100%
GPT‑6.1 TerraOpenAI86%
Fable 5.1Anthropic79%
GPT‑6 SolOpenAI79%
Opus 5Anthropic79%
Opus 5.5Anthropic79%
GPT‑5.6 SolOpenAI71%
Sonnet 5.5Anthropic71%
GPT‑5.6 TerraOpenAI64%
GPT‑6 LunaOpenAI64%
Opus 4.8Anthropic64%
Haiku 4.5Anthropic57%
Sonnet 5Anthropic57%
GPT‑5.6 LunaOpenAI50%
Haiku 5.5Anthropic43%
What models get wrong

They trust what the portal accepts, miss the advisor's course, and read commitments loosely

Catching errors

  1. 69% kept a starting course whose prerequisite the student hadn't passed, which the portal had registered without objection.

Unstated requirements

  1. 63% didn't register the course the advisor's note describes by content, and 75% closed too few degree requirements.

  2. 50% kept the starting class that begins before the car park opens, and 44% had a day on campus longer than the permit covers.

Stateful tool use

  1. 44% left a transfer too short to walk.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 640,000 tokens of input and output.

What's in it

The portal and its pages, the agent harness, the grader and its self‑test, and an answer key with a scripted reference.

Format

Packaged as a Harbor task on request; the portal and grader are standard‑library Python, and the harness drives the site in Chromium.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.