Visual spatial understanding
In these jobs, the sizes, positions and shapes that decide the result are only in pictures. Doing them well means measuring from what a photo shows, reading shapes exactly, and getting positions right.
Three things an agent has to get right
Measure from the picture
Sizes and distances have to be read from what a picture shows, such as a floor tile, a door or a shadow.
Read the details exactly
Look‑alikes differ by colour, size or legs, and flat‑pack panels and fittings have to be counted, so small details decide it.
Get positions right
Pieces have to stand where the photos show, face the right way, and keep clear of door swings and heaters.
They mismeasure, misread details, or put things in the wrong place
Measured wrong
Room furnishing from photos: only 17% of attempts got the room's width right; most rounded the room to whole or half floor tiles, too narrow, with the ceiling too low.
Flood‑claim replacement from dated photos: 63% of attempts bought the loveseat a size smaller than the one that stood in the room before the flood.
Details misread
Flat‑pack assembly manuals: every attempt miscounted the screws and fittings on every piece, and the best found under a third of the real joins.
Flood‑claim replacement from dated photos: 96% of attempts bought the wrong bench, most often in a lighter colour than in the photos.
Misplaced
Room furnishing from photos: no attempt placed every piece within 30 cm of where it stands and facing the right way; attempts that finished got between 1 and 16 of the 17 pieces in place.
Family move‑in from listing photos: in 92% of attempts, the layout stood a piece on a heating register or the heater, which show only in the listing photos.
Partial credit for most attempts; the only full solves are finished race laps
Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.
Hover a dot for its model and reward.
How each environment tests it
| Environment | How it tests this | Checks | Models | Best |
|---|---|---|---|---|
| Flat‑pack assembly manuals with 3D views from unlabelled photosHousehold | Details. From unlabelled photos, the agent counts each piece's panels and fittings, works out how they join, and shows the assembly in 3D. | 173 | 7 | 63% |
| Room furnishing from photos: measure, match look‑alikes, order and planFurnishing service | Measuring. The agent measures the room from six photos, with floor tiles and a door as the only scale, and places 17 pieces in it. | 28 | 8 | 80% |
| Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑upInsurance replacement | Details. Photos from three dates show the rooms changing before the flood, and each like‑for‑like replacement has to match its colour, size and base. | 25 | 8 | 85% |
| Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layoutsMove management | Positions. The listing photos show the steps and heaters, the plans the doors and their swings, and every room has to fit around them. | 45 | 8 | 92% |
| Row‑house rooftop solar designed and quoted from plans, imagery and photosHome solar | Measuring. The roof and the neighbours that shade it are modelled from overhead images, phone photos and plans, with heights read from shadows. | 15 | 6 | 99% |
| Baseball batting by a bimanual robot from batter‑eye imagesVisual robot manipulation | Measuring. Each pitch's position and speed have to be judged from a batter's‑eye image alone, with no numbers given. | Points | 4 | 2 points |
| Four‑car race driven from cockpit images on a live physics clockMotor racing | Positions. Each driver is given only its own cockpit image to judge its place on the road, its heading and the next bend. | 1 | 4 | 100% |
| Four‑car race on a live physics clock with lasting crash damageMotor racing | Positions. The same cockpit view as the first race, where each misjudged bend can now damage the car for good. | 1 | 4 | 100% |
Batting is scored in points, and a race all or nothing. Each environment's datasheet shows how its checks become the reward.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials