AsapiThe AI Agent Trials
Asapi / Catalogue / Tea‑factory ops chat: speak or stay silent
Environment datasheet · Small business

Tea‑factory ops chat: speak or stay silent

ID tb‑tasks/leafline‑tea‑factory · revision 2 · results as of 17 November 2026

In a black‑tea factory's operations chat, the agent reads the guide and weeks of history once and keeps short notes. Then, four times a working day, with only its notes, the guide and the messages since, it stays silent or tells the person responsible a setting is wrong.

Checks
71, 35 must‑speak, 22 must‑stay‑silent, 14 optional, over two scored scenarios
Tested
6 models, one attempt each
Best
53% (Opus 5)
Median
11%
Full solves
0 of 6
Median attempt
172 min, 665 tool calls
What it tests

Speak up or stay silent, four times a working day, from a few pages of notes

Long‑horizon work

Weeks of chat must fit into 16 KB of notes, and later check points have only those, the guide and the new messages.

When to act or escalate

35 check points need the agent to speak up and 22 to stay silent, and wrong interruptions are taken off the catches.

Catching errors

Wrong settings are mixed with correct ones, many of which look wrong against the guide but follow the factory's proven practice.

Unstated requirements

The factory's own settings practice appears only in the past chat, and sometimes contradicts the guide.

How it works

A guide, weeks of chat, 16 KB of notes and 71 check points

The agent's world

A shell in a container, one fresh session at setup and at each check point, and a validator that checks a reply's format only.

What persists

Only the setup notes: the history is removed after setup, and changes made at one check point are lost before the next.

The chat

A few hundred pre‑written messages per scenario, with lab results that come back days later; nothing the agent posts changes what follows.

The reward

Per scenario, the share of catches minus the share of must‑stay‑silent points it got wrong, floored at 0; never posting scores 0.

Results

Attempts scored 0–53%, and for two, wrong calls cancelled out every catch

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each11%0 of 6
ModelReward
Opus 5Anthropic53%
Opus 5.5Anthropic37%
GPT‑6 AstraOpenAI19%
DeepSeek V4.1 FlashDeepSeek2%
GPT‑6 LunaOpenAI0%
GPT‑6.1 SolOpenAI0%

Opus 5 also cleared the task's own pass mark in both scored scenarios, which asks for 50% or more and silence at three in four or more of the points where the crew was right.

What models get wrong

They correct a crew that was right, miss problems, and keep a dryer too hot after its repair

Catching errors

  1. At the 16 points where the crew departs from the guide and is right, Opus 5 held back every time and Opus 5.5 13 times; the other four 1 to 8.

When to act or escalate

  1. At the 35 points where a setting was wrong and could still be fixed, the attempts caught 10 to 20.

Long‑horizon work

  1. After the dryer's thermometer was repaired, the crew's usual setting ran too hot; at the 15 points that needed it turned down, the attempts got 3 to 10 right.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 14 million tokens, cached input included.

What's in it

Three scenarios' guide and chat, two of them scored, the check‑point harness and validator, the grader with its answer key, and reference replies.

Format

Our attempts ran in Harbor, each scenario a setup session and one session per check point; other packaging on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.