Multimodal and large‑corpus evidence

The deciding fact is in an image, or one record among thousands

Images that decide

In 12 of the 21, some deciding evidence exists only in images: photographed handwritten logs, a maker's product sheet inside a PDF, a photographed sheet of a week's figures, scanned temperature sheets, phone screenshots, CCTV stills, unit drawings, plant scans, fitters' marked‑up prints, and room and flat‑pack photos.

No tools given

In six of the bakery and brewery office environments, nothing reads documents: no office suite, PDF text tool, OCR or image tools, and no Python packages. The agent installs what it needs or uses its harness's own image viewer. The kitchen‑fire claim ships document libraries and OCR.

Mixed corpora

The office corpora hold 50 to 402 files: PDFs, Word files, spreadsheets, emails, chat exports, logger data, slide decks and photos. One mixes loggers in raw counts, tenths of a degree and Fahrenheit, on clocks that disagree.

Records by the thousand

Three bill reviews hold 82,283 to 98,093 records for 300 households, searched through a text browser with no list of records: tickets, wiki pages, mailboxes of up to 18,370 messages, card transactions, statements. Quoted email text stays folded until expanded.

Feeds and languages

The holiday‑rental pair joins three listing feeds in three formats (1,101 listings), about 61,000 guest reviews in several languages, a licence register of 10,721 rows and map layers; a rules page says which record wins.

PDFs and feeds

The patient‑history compiler reads 4,490 files, among them 2,724 HL7 files, 331 XML documents, 285 claims files and 151 PDFs: 107 reports read by page layout, 22 blank forms, and 22 filled ones whose values sit inside the PDF's structure.

Drawings and photos

Plant drawings come as ten scans at 300 dpi, some faded or hand‑marked, read by cropping and enlarging regions, or as DXF sheets whose connections exist only in geometry; a room and ten flat‑pack pieces exist only as photos.

Sensor streams

The journey planner's phone hook reads every trip's raw accelerometer, gyroscope, magnetometer, barometer, GPS and screen streams and keeps what it chooses within a byte budget.

Sound

The listening piano‑practice app must hear live piano through the browser microphone, follow it and transcribe a recording; checks are decided from the audio it plays, its MIDI exports and screenshots.

Reachable by design

In all three bill reviews, fairness gates show that reasonable searches, with questions where a client can be asked, reach every deciding record and fact, and that the obvious path alone doesn't reach the hidden ones.

The image or record that decides goes unopened

Each count covers every model tested on the environment, with only the request; where a model made several attempts, the line says so.

1. The image left unopened. Packaging proof sign‑off: 8 of 13 attempts, one per model, never opened the maker's product sheet, an image inside an attached PDF that gives a part's true size. 12 of 13 approved a box the tallest cake would almost touch.

2. The bill never opened. Household bill review: 12 of 24 attempts (8 models × 3) asked a company for money its later bill had already credited; none of them had opened that bill. GPT‑6 Astra and GPT‑6.1 Sol, searching company by company, caught it in all 6 of theirs.

3. One record in 82,000. Medical bills review: 16 of 24 attempts (8 models × 3) missed a form the plan needed back, printed only on one claim's statement. GPT‑6 Astra found it in all 3 of its attempts by searching for it; all 3 of Opus 5.5's missed it.

4. Handwriting misjudged. Temperature compliance register: 5 of 12 attempts, one per model, left nothing the grader could read, one saving only pictures of the scans. Of the 7 readable registers, 4, the top three among them, distrusted genuine entries in an unfamiliar hand.

5. Guest against host. Exact holiday‑rental retrieval: where a listing's host and only guest disagree on its direction, 3 of 6 attempts found the guest's word and let the guest win, as the rules say. Opus 5.5 never matched it; Sonnet 5.5 printed the review but didn't act.

6. The export read instead. Scanned P&ID digitisation: 1 of 5 attempts, one per model, built the instrument index from the control system's out‑of‑date export rather than the scans, leaving out 161 instruments, and stopped after 4 of its 90 minutes.

Five full solves in 243 attempts, in four of the 21

Each dot is one attempt with only the request; the line marks the median. Larger dots are full solves.

Hover a dot for its model and reward.

Where each one puts the evidence

Checks: a range where the count varies by attempt. Told the rules: in the claim pack, the waste report, the temperature register and the room furnishing it also gives facts from the files, in all but the register figures the grader checks. A dash: not tested.

What we haven't measured

• Images decide in 12 of the 21, and the room and flat‑pack photos were rendered from 3D models, so they're cleaner than real ones. The bill reviews show every file as text, and the holiday‑rental pair, compiler and journey planner have no images: there, the difficulty is volume and format.
• Our results don't separate failing to install a document or image tool from failing to read what it shows: six of the office environments ship none.
• Shortcuts still pay in the two report environments: a figures‑only report scores 84% in the waste report, and a bare table with its total, one note and a recommendation 97% in the price report.
• In the two holiday‑rental environments, our attempts weren't isolated from the answer key.
• In the piano app, the sound delay bears on up to six build checks and the speed‑ladder checks that need a take with no mistakes, in about 19 of its 21 attempts across versions; its size per attempt isn't proven.