AsapiThe AI Agent Trials
Asapi / Capabilities / Multimodal records
Capability · 23 environments

Multimodal records

The records an agent works from are rarely tidy: mailboxes, spreadsheets, scans, photos and handwriting, each in its own format. Doing well means finding the deciding record, reading what isn't text, and joining sources up.

What it takes

Three things an agent has to get right

Find the record that decides it

The deciding fact can sit in one attachment among hundreds of files, or one record among tens of thousands, and nobody flags it.

Read what isn't text

Some figures, rules and labels exist only in photos, scans or handwriting, and some facts only in phone sensor streams.

Join them up

Sources come in different formats, languages, units and clocks, and the answer often needs two of them read together.

What models get wrong

They miss the deciding record, misread an image, or don't join the sources

The deciding record missed

  1. Packaging proof sign‑off: 77% of attempts never checked a supplier's stated size for a newly swapped part against the maker's product sheet in the mailbox.

Images misread

  1. Photographed P&ID sketches digitised: 80% of attempts put a gauge's connection or a line's end on the wrong branch of a pump group, reading pencil sketches of the plant's piping and instruments.

  2. Furniture warranty claim: 58% of attempts took a label photo of a different chair as the damaged chair's.

Not joined up

  1. P&ID as‑built backlog: 83% of attempts accepted a contractor's claim that a drain was already on the plant drawing, without finding the pipe it describes, and left the drain undrawn.

  2. Multi‑turn holiday‑rental search chat: in half the attempts, the agent read the customers' saved profiles, then searched without the standing needs they hold, such as a top floor or a lift.

Results

Partial credit for nearly every attempt, and few full solves

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

EnvironmentMedianFully solved
Household bill review: card disputes, claims, tracker8 models × 3 attempts86%0 of 24Onboarding bill review with a simulated client8 models × 3 attempts78%0 of 24Medical bills review across two health plans8 models × 3 attempts74%0 of 24Household savings plan from records and family8 models50%0 of 8Packaging proof sign‑off for an absent owner13 models2%0 of 13Monthly waste report in house format13 models64%0 of 13Brewery input‑price report in house format12 models73%0 of 12Contractor invoice audit against shop records13 models44%0 of 13Bakery's freezer‑loss insurance claim pack14 models55%0 of 14Temperature compliance register from messy logger data12 models9%0 of 12Kitchen‑fire contents claim from stale counts, invoices and photos7 models81%0 of 7Listening piano‑practice app: following, transcription, feedback7 models53%0 of 7Offline fault‑finding tablet app for data‑centre cooling units7 models64%0 of 7Transit journey planner and phone telemetry over five field weeks5 models84%0 of 5Furniture warranty claim from a simulated customer's photos8 models × 3 attempts82%0 of 24Multi‑turn holiday‑rental search chat with saved profiles6 models34%0 of 6Exact holiday‑rental retrieval over multilingual multi‑source data6 models88%1 of 6Tea factory rooftop solar and battery design with its business case6 models66%0 of 6Emergency patient‑history compiler from a long spec6 models99%2 of 6Scanned P&ID set digitised into instrument, line and equipment lists5 models99%1 of 5Photographed P&ID sketches digitised into instrument, line and equipment lists5 models99%0 of 5P&ID as‑built backlog from nine teams' records, drafted in DXF6 models97%1 of 6Row‑house rooftop solar designed and quoted from plans, imagery and photos6 models71%0 of 6

Hover a dot for its model and reward.

The 23 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Household bill review: card disputes, claims, trackerHousehold billsFinding. The deciding records sit among about 98,000, including an 18,370‑message mailbox, searched through a text browser.191–262894%
Onboarding bill review with a simulated clientHousehold billsFinding. About 90,000 records for 300 households are searched through a text browser, with no list of what's there.295–320884%
Medical bills review across two health plansHousehold billsFinding. About 82,000 records for 300 households hold problems that sit in single records nobody flagged.287–350888%
Household savings plan from records and familyHouseholdBeyond text. Chats, social posts and side earnings exist only as phone screenshots, cited by 24 of the 51 checks.51885%
Packaging proof sign‑off for an absent ownerSmall businessFinding. The deciding sizes are in box proofs and a maker's product sheet, images inside attachments in three mailbox archives.151384%
Monthly waste report in house formatSmall businessBeyond text. One shop's week of figures survives only in a photo of a paper sheet, and two format rules only in handwriting.431389%
Brewery input‑price report in house formatSmall businessBeyond text. A brew sheet and delivery notes exist only as photos, and the director's markup on last month's report is an image‑only scan.511290%
Contractor invoice audit against shop recordsSmall businessBeyond text. Photos of engineers' handwritten log pages are evidence for 31 of the invoice's 44 lines, among 171 files.1071361%
Bakery's freezer‑loss insurance claim packSmall businessJoining. The deciding facts are spread over 111 files in nine formats, from emails and chat exports to logger files, one in UTC.601483%
Temperature compliance register from messy logger dataSmall businessBeyond text. Scanned handwritten sheets and phone photos hold readings found nowhere else, beside logger files in mixed units and clocks.1031240%
Kitchen‑fire contents claim from stale counts, invoices and photosSmall businessBeyond text. Photos, CCTV stills and scans have no text: the oven's rating plate, for one, decides its replacement.29799%
Listening piano‑practice app: following, transcription, feedbackHouseholdBeyond text. The app must hear live piano through the microphone and follow it, checked against recordings and a hidden reference transcription.64758%
Offline fault‑finding tablet app for data‑centre cooling unitsFacilitiesBeyond text. The drawings are PDFs read only by looking, and each test point the app names must match that unit's drawing.79787%
Transit journey planner and phone telemetry over five field weeksSoftware for a transit operatorBeyond text. Some facts show only in phone sensor streams, which the agent's code must shrink to a small daily budget.81598%
Furniture warranty claim from a simulated customer's photosCustomer serviceBeyond text. Each chair's version, order and damage exist only in photos, viewed with the agent's own image tool.56895%
Multi‑turn holiday‑rental search chat with saved profilesCustomer serviceJoining. Answers must join each customer's saved profile with three listing feeds in different formats and languages, a register and map layers.25688%
Exact holiday‑rental retrieval over multilingual multi‑source dataCustomer serviceJoining. Three feeds in three formats and several languages hold 1,101 listings, and some requirements appear only in guest reviews.206100%
Tea factory rooftop solar and battery design with its business caseEnergy engineeringJoining. The design draws on 91 files in their sources' own formats, from interval data and bills to rotated plans and site photos.18692%
Emergency patient‑history compiler from a long specHealth records softwareJoining. Evidence arrives as 4,490 files: message feeds, clinical documents, claims files and PDF reports read by layout.9196100%
Scanned P&ID set digitised into instrument, line and equipment listsEngineering recordsBeyond text. Every row is read from ten scanned sheets, some faded or hand‑marked, where no tag is readable at whole‑sheet size.3,8365100%
Photographed P&ID sketches digitised into instrument, line and equipment listsEngineering recordsBeyond text. Every row is read from 17 photographed pencil pages in the drafter's own symbols, with look‑alike digits to tell apart.3,836599%
P&ID as‑built backlog from nine teams' records, drafted in DXFEngineering recordsJoining. Changes are traced through about 700 records in nine teams' formats and drawn onto nine CAD sheets.656100%
Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solarBeyond text. The roof and its neighbours exist only in overhead captures, phone photos and plans, with heights read from shadows.15699%

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.