Course registration against rules found in portal

The request says commitments exist; the records hold every rule

Role

Acting for a second‑year student in registration week, on the university's portal.

Request

Sort out the student's autumn registration so classes fit around other commitments already booked for the term, which are listed in the portal, without paying more than needed.

Deliverable

The registration the portal holds when the agent stops. The closing message is recorded but not graded.

What the reward depends on

Turning a work shift, a training slot and a parking permit into times to avoid; the scholarship's credit floor, with no waitlisted seat; no clashes and walkable transfers; the course the advisor's note describes; enough degree requirements closed; prerequisites met.

Where the requirements are

Only in the student's records on the portal: term commitments, scholarship terms, the degree audit, the student record, an advisor's note and the campus map. No page states a rule as a rule, and the starting timetable already breaks several.

An 87‑page portal that registers anything and flags only some of it

Interface

A Python harness loads a live portal in headless Chromium and gives the model each page as numbered lines of text; the model answers with one action a turn. No container, terminal, files or web: any address off the portal is refused.

Portal

87 pages at the start: a catalogue search that lists nothing until searched, 51 course pages (13 departments, 108 sections, 13 full), the cart, the schedule, the degree audit, the student's records, an advisor's note and a campus map with 28 places.

Actions

Click, type, select, go to a path on the site, go back, or stop with a closing message. Adding a section only fills a cart until the registration is submitted; removing a registered section drops it at once.

What the site allows

It blocks nothing: it registers clashes, full sections (as a waitlist place) and courses without their prerequisites. It flags clashes, full sections and walks too long for the gap, but not a missing prerequisite.

Model calls

Each reply is one fresh Claude Code call with Claude Code's own tools off. The harness passes the conversation so far, with pages older than 40 turns replaced by a placeholder.

Stops

Four replies in a row with no valid action, or one action repeated five times, end an attempt; otherwise it ends when the agent stops or reaches 250 replies.

Time and people

A fixed in‑world date; no clock runs and nobody answers. The registration changes only through the agent's actions.

14 script checks on the portal's record; an empty registration meets 11

Structure

14 yes/no checks, one point each: 6 on booked commitments, 2 on the scholarship, 2 on a workable timetable, 4 on degree progress. Reward = checks met / 14; a pass is all 14.

Who decides

Scripts alone, on the portal's own record of the registered sections when the agent stops; sections left in the cart don't count. No LLM judge, no penalties, and the closing message isn't read.

Rebuilt and replayed

An offline grader rebuilds each final registration in a fresh portal and reaches the same verdict. Its self‑test passes the answer key, fails the starting registration and regrades every attempt to its recorded checks.

Reference and floor

A scripted reference scores 100%, and six registrations the catalogue allows meet all 14. The starting registration meets 7 (50%); an empty one meets 11 (79%), since 11 checks only look for broken rules.

The three best attempts matched an empty registration's 79%

Three of four attempts kept every commitment and scored 79%, what an empty registration scores: each missed the advisor's course, the degree requirements and a prerequisite. The fourth scored 57%. No attempt fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model, by Anthropic models only: the harness reached models through Claude Code. Attempts by four models from other makers ran on the version before the 12 October fixes and aren't on our board.

There is no version with a hint or with the rules written in.

What the models got wrong

Counted over the 4 models, one attempt each, with only the request.

1. Silent acceptance trusted. 4 of 4 kept a starting course whose prerequisite the student hadn't passed, which the portal had registered without objection; none opened its page, and 3 added the missing prerequisite in the same term.

2. The deciding record unread. None of the 4 opened the advisor's note, so all missed the course it describes by content and closed too few requirements; only one opened the degree audit.

3. Swapped away. One attempt held the advisor's course, then dropped it to fix two tight walks caused by starting classes it kept, one of them a course the student wasn't ready for.

4. One problem fixed, the rest left. 1 of 4 treated the job as two missing credits: it kept a class before the car park opens, a day longer than the permit allows and an unwalkable transfer, while its closing message said the day fit.

What an attempt costs

Length

The median attempt took 3.5 minutes and 24 actions (10 to 30 across the 4), and about 330,000 tokens of input and output; the harness records no cache figures.

Cost

The median attempt cost $2.93 at list prices, from $0.34 to $9.46 across the 4; all 4 came to $16. That's the agent model only, and nothing else in the environment calls a model.

What you license

With a licence

Revision 2: the portal and its pages, the agent harness, the grader and its self‑test, the answer key with a scripted reference and a valid registration, and the grader's report on each of the 4 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today the portal and grader are standard‑library Python, and the harness drives the site with Playwright and Chromium, reaching models through the Claude Code CLI.

Revisions

Revision 2 follows an audit on 9 October: the records say how the permit's hours count, where training is, that it ends earlier and that shift days are barred; the training check counts the walk to the pool and back; the request asks to pay no more than needed.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Its job page on The AI Agent Trials, our public benchmark, describes the job, its main pitfalls and each model's results in plain language, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• An empty registration meets 11 of 14 checks, as much as the best attempt, since 11 checks only look for broken rules. The owner kept this scoring: only 100% counts as a pass.
• Leaving the starting registration as it is meets 7 of 14 checks (50%).
• Only 4 attempts, one per model, all by Anthropic models, so we haven't measured how much attempts vary or how other makers' models do on this revision.
• The credit check counts a waitlisted class's credits, though the scholarship counts them only once a seat is confirmed. No attempt held a waitlisted class.
• The checks read only the final registration, so a closing message that misreports it costs nothing; one attempt said the day fit the permit when it didn't.
• No red‑team attempts yet.