Tea‑factory ops chat: speak or stay silent

Read the history once, then speak or stay silent

Role

An operations assistant in the crew's group chat that controls nothing and can only post.

Request

At setup, study the guide and the history and leave notes, since the history goes away afterwards. At each check point, post a correction to whoever is responsible when a setting doesn't suit the leaf or a step is about to be skipped and there's still time; otherwise say nothing.

Deliverable

At setup, notes: text only, at most 16 KB, at most a fifth of lines copied from the chat. At each check point, one reply: empty, or a short message with typed actions naming equipment, setting, level, direction and the troughs or lots.

What the reward depends on

Learning the factory's own practice from the chat, including where it departs from the guide and is right; speaking up in time, naming every trough affected; staying silent when the crew is right; noticing when a repair makes an old habit wrong.

Where the requirements are

Only in the chat: the factory's rules are never written down, and some contradict the generic guide. The request says the factory's way wins where the history shows it works.

A setup session, then 106 check‑point sessions with only the notes

Interface

A shell in a Docker container (Ubuntu, Python), one fresh session at setup and at each check point, driven by Claude Code, Codex CLI or Hermes in our attempts. A validator checks a reply's format, not its content.

Sessions

Three scenarios, each a setup session (60 minutes) with the guide and 17 to 20 days of chat, then 35 or 36 check points (10 minutes each) at 08:00, 11:00, 14:00 and 17:00 on working days: 109 sessions per attempt.

What persists

Only the setup notes. The verifier checks them against the setup rules, then removes the history; before each check point it restores the notes as saved, so changes made at one check point are lost.

Check‑point view

The guide, the notes, the messages since setup (one file a day), today's messages so far and the check point's time.

The chat

A few hundred short messages per scenario: withering checks, leaf wetness, tasting and lab results that come back days later, weather, settings and small talk. Nothing the agent posts changes what follows, and nobody replies.

The crew

A withering master about to retire, a dryer operator going on leave, a new hire who knows only the guide, and stand‑ins. No model plays them: every message was written in advance.

The guide

A generic tea‑processing guide with the reply format, sound in general but wrong or silent on parts of this factory's practice. Following it scores below silence.

What matters

Two settings: the dryer's inlet heat and the troughs' overnight hot air. Partway through, a piece of equipment is repaired and one of the crew's long habits stops being right.

Behind the chat

A hidden plant model, not in the package, generated the events behind every message and each check point's answer from counterfactual sweeps. The package holds the pre‑written chat, the answer key and the grader, root‑only.

Speaking and staying silent weigh the same

Structure

106 checks, one per check point over three scenarios (36, 35 and 35): 47 where the agent must speak; 42 where it must stay silent, 29 of them where the crew departs from the guide and is right; and 17 where either is fine.

Must speak

Every expected action must match (equipment, setting, allowed level, direction), every trough or block it should fix must be named, and no action may contradict the key. Levels are broad bands, not exact numbers; trough names match however written.

Must stay silent

Any action fails, except a correct 'leave it' answer to a question. Where either is fine, silence passes and speech must not contradict the key.

Formula

Per scenario, the mean of the must‑speak and must‑stay‑silent pass rates; reward = the mean over the three scenarios, from 0 to 1. The 17 optional points are recorded but outside the reward.

Gates

Notes that break the setup rules end the scenario with reward 0. The task's own pass, kept beside the reward, needs a balanced rate of 0.80 and silence at 0.75 of the right departures.

Who decides

Scripts match each reply against the plant model's labels; no LLM judge, and nothing is replayed. All six attempts were regraded on 13 October from their saved replies, after a 9 October audit found four grading problems; every score fell by 3 to 10 points.

Self‑test

The answer key scores 1 in Harbor. Never posting scores exactly 50% on every scenario.

No attempt passed a scenario; three scored below silence

With only the request, the 6 attempts, one per model, scored 40–67%, and none passed a scenario. Never posting scores 50%, and three attempts scored below it. The closest, Opus 5, missed the task's own pass on one scenario by about two points.

Each dot is one model's attempt; the line marks the median.

One attempt per model, each covering the three scenarios; one version of the request.

Before the 13 October regrade, Opus 5.5 ranked first and each Opus model passed one scenario; under the corrected grader, Opus 5 leads and no attempt passes one.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request, on the 106 check points each faced.

1. Correcting a crew that was right. At the 29 points where the crew departs from the guide and is right, the two best attempts stayed silent 21 and 20 times, the other four 10 or 11. All 6 told the crew to change a correct setting at least 8 times.

2. Missed problems. At the 47 points where a setting was wrong and could still be fixed, the attempts caught 14 to 27; 6 of the 47 were caught by none of the 6.

3. Partial corrections. A correction counts only if it names every trough that needs it; naming only some cost every attempt between 3 and 9 check points.

4. An old habit after a repair. After a repair made one of the crew's settings wrong, 3 of 6 attempts kept defending the old setting; two changed it, often for only some troughs, and one often missed it.

5. Below doing nothing. 3 of 6 attempts scored under the 50% that never posting earns: DeepSeek V4.1 Flash 44%, GPT‑6.1 Sol 41% and GPT‑6 Luna 40%.

What an attempt costs

Length

With only the request, the median attempt took 172 minutes of agent time and 665 tool calls over its 109 sessions (385 to 2,352 across the 6), and about 14 million tokens, cached input included.

Cost

Recorded for all 6. Opus 5's attempt cost $49.97 and Opus 5.5's $21.69 at list prices; Harbor's figures are $48.67 for GPT‑6 Astra and $8.77 for GPT‑6.1 Sol. GPT‑6 Luna's $0.60 uses a derived price and DeepSeek V4.1 Flash's $1.97 is its agent's own estimate, so we give no median.

What it covers

The agent model only; nothing inside the environment calls a model.

What you license

With a licence

The three scenarios' guide and chat, the check‑point harness and validator, the grader with its answer key, the reference replies, and the grader's report on each of the 6 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor on EC2, each scenario a multi‑step task of a setup session and one session per check point.

Levels

Level 1, tested here, has two live settings and five factory rules. Levels 2 to 4, with up to ten settings and about 13 rules, are built and pass their reference replies, but no model has attempted them; one level‑4 rule fails the builder's own learnability check.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The chat, answer key and grader aren't published.

What we haven't measured, and what's uneven

• Posting nothing scores 50%, since the reward averages speaking when needed and staying silent; three of the six attempts scored below it.
• The labels come from a hidden plant model that isn't in the package, and the builder's check that every rule shows in the history with outcomes can't be rerun here. The 9 October audit found three cases where the key took one of two reasonable readings; both are now accepted.
• Four attempts ran on an earlier build that didn't clear scratch folders between sessions. Opus 5 left copies of the history there at setup; they were removed by hand during its attempt, and no check‑point session used them.
• DeepSeek V4.1 Flash searched the web for the task at two check points and found nothing useful; its score stands. Six of its check points timed out with no reply and count as failed.
• One attempt per model, so we haven't measured how much attempts vary here.
• Each model ran in its maker's harness (Hermes for DeepSeek V4.1 Flash), so a score is for the pair.
• No red‑team attempts yet.