The tea factory's chat

Learn a tea factory's own practice from weeks of crew chat, keep it in short notes, then speak up only when a setting is about to go wrong.

Setting: Tea factory, operations chat
Deliverable: Notes at setup, then silence or a correction four times a day

Conflicting sourcesUnstated requirementsMessy documents

Models

ran the standard setup

67%

Best score

Opus 5

49%

Median score

across those models

2.9 h

Typical run

median time, standard setup

The task

A made-up black-tea factory runs on its crew's know-how: a withering master about to retire, a dryer operator going on leave, a new hire who knows only the textbook. The agent joins their group chat as an assistant that controls nothing and can only post. It reads weeks of chat and a generic processing guide once, and keeps what it learns in small notes. Then the history is taken away: four times a day it sees only its notes and the latest messages, and stays silent or writes a short correction.

The prompt only asks it to flag settings that are wrong. The real test is telling the crew's deliberate habits from real mistakes: working out where the factory differs from the guide and why, keeping that in a few pages, and holding back when the crew is right.

What it tests

Learning a practice from examples

No rule is stated as a rule; each shows up as episodes in the chat, with the outcomes coming back days later.

Weighing a guide against experience

The generic guide is right in general but not for this factory, and only the chat shows where.

Knowing when to stay quiet

Speaking up when nothing is wrong counts against the agent as much as missing a real mistake.

Remembering through notes

Each time it is called, the agent has only what it chose to write down.

Challenges

1A guide that does not fit

The written guide is sound in general but wrong or silent on this factory's practice, as generic manuals often are.

2Experts who depart from the guide on purpose

A veteran does things differently and is right, and correcting him is a mistake.

3A rule that changes partway

A piece of equipment is repaired during the weeks the agent watches, and an old habit stops being right.

4A tempting explanation

A bad result has an obvious cause that the outcomes in the chat show is not the real one.

5Closed book

The history is gone after setup, and the notes are capped in size and may copy little of the chat.

Results

Model	Standard	Time (standard)	Cost (standard)
Opus 5	67%	201 min	$49.97
Opus 5.5	63%	106 min	$21.69
GPT-6 Astra	54%	156 min	$48.67
DeepSeek V4.1 Flash	44%	381 min	$1.97
GPT-6.1 Sol	41%	163 min	$8.77
GPT-6 Luna	40%	181 min	$0.60

One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

Opus 5 led at 66.6% and Opus 5.5 followed at 62.7%; no model passed a version of the factory. The runs were rescored on 13 October 2026 under stricter, corrected scoring; before that, Opus 5.5 was first and both Opus models passed one version.
Holding back was the commonest failure. Opus 5, Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash, GPT-6.1 Sol and GPT-6 Luna all spoke up where the crew was right, at 8 moments for Opus 5 and up to 19 for GPT-6 Astra, DeepSeek V4.1 Flash and GPT-6.1 Sol.
The change partway through caught Opus 5.5, Opus 5 and DeepSeek V4.1 Flash, which kept defending the crew's old setting after the repair; GPT-6 Astra and GPT-6.1 Sol turned it down, but often for only some of the troughs that needed it.
Every model at times named only some of the troughs a correction should fix, from 3 moments for DeepSeek V4.1 Flash to 9 for Opus 5.5.
GPT-6 Astra and DeepSeek V4.1 Flash took the tempting explanation of a bad batch over the outcomes in the chat, and GPT-6.1 Sol overrode the crew with an untested recipe of its own.
DeepSeek V4.1 Flash (44.0%), GPT-6.1 Sol (41.4%) and GPT-6 Luna (40.1%) scored below the 50.0% that never posting would earn. GPT-6 Luna wrote the thinnest notes, and did best, 54.2%, on the one version where they captured the crew's habit.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Simulated work chat of a made-up black-tea factory: the agent is an operations assistant in the crew's group chat that controls nothing and can only post. The chat and the answer for every check point were written in advance by a hidden plant model, which is not in the container or in this repository. Three versions of the factory (seeds 0, 1 and 2), each its own run
Agent's interface: A terminal in a fresh Docker container (Ubuntu 24.04, Python 3; 2 CPUs, 4 GB), one fresh session at setup and at every check point as an unprivileged user, driven by the agent's own command-line tool through harbor: Claude Code, Codex CLI or hermes
Documents: Per version: 1 Markdown guide (/app/docs/plant_operations.md, generic tea processing with the reply format) and 17 to 20 TXT chat files, one per working day, in /app/thread/history/; 57 files over the three versions. The history is removed after setup. At each check point the agent gets TXT files of the messages since setup (one per day), today's messages so far and the check point's date and time
Applications: One command, validate_reply, which checks a reply file's format, not its content. No other apps
Web: Not needed: nothing the task asks about is online. The container can reach the internet. Claude Code (Opus 5.5, Opus 5) had its web-fetch tool on, with no web-search tool offered, and never used it; Codex (GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna) ran with web search turned off and used no web tool; hermes (DeepSeek V4.1 Flash) had no web tools, but at two check points it searched the web from the terminal and found nothing useful
People: None who answer. The crew's messages were written in advance by the plant model; nobody reads or replies to the agent's posts
Time: Fixed check points at 08:00, 11:00, 14:00 and 17:00 of each evaluation day: 36 in version 0, 35 in versions 1 and 2. At each, the agent sees the chat up to that moment
Actions: At setup, write notes in /workspace; at each check point, write one reply file. Nothing the agent writes changes the chat or the factory
Deliverable: At setup: notes in /workspace, only text files, at most 16,384 bytes in all, with no more than a fifth of the lines copied from the chat. At each check point: one JSON file, /app/out/reply.json, with a message and a list of actions (equipment, setting or step, target level, direction, and the troughs or lots it applies to), or empty for silence

← PreviousThe tea factory's cash desk All 34 tasks Next →The line fit-out