Multimodal records
The records an agent works from are rarely tidy: mailboxes, spreadsheets, scans, photos and handwriting, each in its own format. Doing well means finding the deciding record, reading what isn't text, and joining sources up.
Three things an agent has to get right
Find the record that decides it
The deciding fact can sit in one attachment among hundreds of files, or one record among tens of thousands, and nobody flags it.
Read what isn't text
Some figures, rules and labels exist only in photos, scans or handwriting, and some facts only in phone sensor streams.
Join them up
Sources come in different formats, languages, units and clocks, and the answer often needs two of them read together.
They miss the deciding record, misread an image, or don't join the sources
The deciding record missed
Packaging proof sign‑off: 77% of attempts never checked a supplier's stated size for a newly swapped part against the maker's product sheet in the mailbox.
Images misread
Photographed P&ID sketches digitised: 80% of attempts put a gauge's connection or a line's end on the wrong branch of a pump group, reading pencil sketches of the plant's piping and instruments.
Furniture warranty claim: 58% of attempts took a label photo of a different chair as the damaged chair's.
Not joined up
P&ID as‑built backlog: 83% of attempts accepted a contractor's claim that a drain was already on the plant drawing, without finding the pipe it describes, and left the drain undrawn.
Multi‑turn holiday‑rental search chat: in half the attempts, the agent read the customers' saved profiles, then searched without the standing needs they hold, such as a top floor or a lift.
Partial credit for nearly every attempt, and few full solves
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Household bill review: card disputes, claims, trackerHousehold bills | Finding. The deciding records sit among about 98,000, including an 18,370‑message mailbox, searched through a text browser. | 191–262 | 8 | 94% |
| Onboarding bill review with a simulated clientHousehold bills | Finding. About 90,000 records for 300 households are searched through a text browser, with no list of what's there. | 295–320 | 8 | 84% |
| Medical bills review across two health plansHousehold bills | Finding. About 82,000 records for 300 households hold problems that sit in single records nobody flagged. | 287–350 | 8 | 88% |
| Household savings plan from records and familyHousehold | Beyond text. Chats, social posts and side earnings exist only as phone screenshots, cited by 24 of the 51 checks. | 51 | 8 | 85% |
| Packaging proof sign‑off for an absent ownerSmall business | Finding. The deciding sizes are in box proofs and a maker's product sheet, images inside attachments in three mailbox archives. | 15 | 13 | 84% |
| Monthly waste report in house formatSmall business | Beyond text. One shop's week of figures survives only in a photo of a paper sheet, and two format rules only in handwriting. | 43 | 13 | 89% |
| Brewery input‑price report in house formatSmall business | Beyond text. A brew sheet and delivery notes exist only as photos, and the director's markup on last month's report is an image‑only scan. | 51 | 12 | 90% |
| Contractor invoice audit against shop recordsSmall business | Beyond text. Photos of engineers' handwritten log pages are evidence for 31 of the invoice's 44 lines, among 171 files. | 107 | 13 | 61% |
| Bakery's freezer‑loss insurance claim packSmall business | Joining. The deciding facts are spread over 111 files in nine formats, from emails and chat exports to logger files, one in UTC. | 60 | 14 | 83% |
| Temperature compliance register from messy logger dataSmall business | Beyond text. Scanned handwritten sheets and phone photos hold readings found nowhere else, beside logger files in mixed units and clocks. | 103 | 12 | 40% |
| Kitchen‑fire contents claim from stale counts, invoices and photosSmall business | Beyond text. Photos, CCTV stills and scans have no text: the oven's rating plate, for one, decides its replacement. | 29 | 7 | 99% |
| Listening piano‑practice app: following, transcription, feedbackHousehold | Beyond text. The app must hear live piano through the microphone and follow it, checked against recordings and a hidden reference transcription. | 64 | 7 | 58% |
| Offline fault‑finding tablet app for data‑centre cooling unitsFacilities | Beyond text. The drawings are PDFs read only by looking, and each test point the app names must match that unit's drawing. | 79 | 7 | 87% |
| Transit journey planner and phone telemetry over five field weeksSoftware for a transit operator | Beyond text. Some facts show only in phone sensor streams, which the agent's code must shrink to a small daily budget. | 81 | 5 | 98% |
| Furniture warranty claim from a simulated customer's photosCustomer service | Beyond text. Each chair's version, order and damage exist only in photos, viewed with the agent's own image tool. | 56 | 8 | 95% |
| Multi‑turn holiday‑rental search chat with saved profilesCustomer service | Joining. Answers must join each customer's saved profile with three listing feeds in different formats and languages, a register and map layers. | 25 | 6 | 88% |
| Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer service | Joining. Three feeds in three formats and several languages hold 1,101 listings, and some requirements appear only in guest reviews. | 20 | 6 | 100% |
| Tea factory rooftop solar and battery design with its business caseEnergy engineering | Joining. The design draws on 91 files in their sources' own formats, from interval data and bills to rotated plans and site photos. | 18 | 6 | 92% |
| Emergency patient‑history compiler from a long specHealth records software | Joining. Evidence arrives as 4,490 files: message feeds, clinical documents, claims files and PDF reports read by layout. | 919 | 6 | 100% |
| Scanned P&ID set digitised into instrument, line and equipment listsEngineering records | Beyond text. Every row is read from ten scanned sheets, some faded or hand‑marked, where no tag is readable at whole‑sheet size. | 3,836 | 5 | 100% |
| Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering records | Beyond text. Every row is read from 17 photographed pencil pages in the drafter's own symbols, with look‑alike digits to tell apart. | 3,836 | 5 | 99% |
| P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering records | Joining. Changes are traced through about 700 records in nine teams' formats and drawn onto nine CAD sheets. | 65 | 6 | 100% |
| Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solar | Beyond text. The roof and its neighbours exist only in overhead captures, phone photos and plans, with heights read from shadows. | 15 | 6 | 99% |
Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials