Tea‑factory ops chat: speak or stay silent
ID tb‑tasks/leafline‑tea‑factory · revision 2 · results as of 17 November 2026
In a black‑tea factory's operations chat, the agent reads the guide and weeks of history once and keeps short notes. Then, four times a working day, with only its notes, the guide and the messages since, it stays silent or tells the person responsible a setting is wrong.
- Checks
- 71, 35 must‑speak, 22 must‑stay‑silent, 14 optional, over two scored scenarios
- Tested
- 6 models, one attempt each
- Best
- 53% (Opus 5)
- Median
- 11%
- Full solves
- 0 of 6
- Median attempt
- 172 min, 665 tool calls
Speak up or stay silent, four times a working day, from a few pages of notes
Long‑horizon work
Weeks of chat must fit into 16 KB of notes, and later check points have only those, the guide and the new messages.
When to act or escalate
35 check points need the agent to speak up and 22 to stay silent, and wrong interruptions are taken off the catches.
Catching errors
Wrong settings are mixed with correct ones, many of which look wrong against the guide but follow the factory's proven practice.
Unstated requirements
The factory's own settings practice appears only in the past chat, and sometimes contradicts the guide.
A guide, weeks of chat, 16 KB of notes and 71 check points
The agent's world
A shell in a container, one fresh session at setup and at each check point, and a validator that checks a reply's format only.
What persists
Only the setup notes: the history is removed after setup, and changes made at one check point are lost before the next.
The chat
A few hundred pre‑written messages per scenario, with lab results that come back days later; nothing the agent posts changes what follows.
The reward
Per scenario, the share of catches minus the share of must‑stay‑silent points it got wrong, floored at 0; never posting scores 0.
Attempts scored 0–53%, and for two, wrong calls cancelled out every catch
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5Anthropic | 53% |
| Opus 5.5Anthropic | 37% |
| GPT‑6 AstraOpenAI | 19% |
| DeepSeek V4.1 FlashDeepSeek | 2% |
| GPT‑6 LunaOpenAI | 0% |
| GPT‑6.1 SolOpenAI | 0% |
Opus 5 also cleared the task's own pass mark in both scored scenarios, which asks for 50% or more and silence at three in four or more of the points where the crew was right.
They correct a crew that was right, miss problems, and keep a dryer too hot after its repair
Catching errors
At the 16 points where the crew departs from the guide and is right, Opus 5 held back every time and Opus 5.5 13 times; the other four 1 to 8.
When to act or escalate
At the 35 points where a setting was wrong and could still be fixed, the attempts caught 10 to 20.
Long‑horizon work
After the dryer's thermometer was repaired, the crew's usual setting ran too hot; at the 15 points that needed it turned down, the attempts got 3 to 10 right.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 14 million tokens, cached input included.
What's in it
Three scenarios' guide and chat, two of them scored, the check‑point harness and validator, the grader with its answer key, and reference replies.
Format
Our attempts ran in Harbor, each scenario a setup session and one session per check point; other packaging on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials