Baseball batting by a bimanual robot from batter‑eye images
ID llarms/task11 · revision 2 · results as of 10 November 2026
In a physics simulation running at a 50th of real speed, a two‑handed robot holds a bat. Seeing only the batter's camera view, the agent sets shoulder and elbow targets to hit ten mixed pitches, scoring for contact and for fair carry.
- Checks
- Points, 1 for contact plus 1 per metre of fair carry, over 10 pitches
- Tested
- 4 models, one attempt each
- Best
- 2 points (GPT‑6.1 Sol)
- Median
- 1 point
Ten pitches hit by a two‑armed robot, judged from the batter's eye
Visual spatial understanding
Each pitch's position and speed have to be judged from a batter's‑eye image alone, with no ball coordinates given.
Robot control
Against ten pitches of four kinds, the agent swings using only shoulder and elbow targets, while the world keeps moving.
A batter's‑eye camera, eight joint targets and a world that never pauses
The agent's world
Four tools, look, start, move and finish, through the model's usual coding agent, with its shell and web tools off.
What it sees
One 960 by 540 image from the batter's eyes per look or move, with the score and timing; no ball coordinates or joint readings.
Control
Eight joint targets, three at each shoulder and one at each elbow; the world runs on at a 50th of real speed, never pausing.
The reward
Over ten pitches, a contact earns a point, and a fair hit one more per metre of carry to its first bounce; there's no maximum.
The four complete attempts made four contacts, every one foul
Each dot is one attempt, and the line marks the median.
| Model | Points |
|---|---|
| GPT‑6.1 SolOpenAI | 2 points |
| GPT‑6 AstraOpenAI | 1 point |
| GPT‑6 LunaOpenAI | 1 point |
| GPT‑6.1 TerraOpenAI | 0 points |
Three more attempts stopped before the tenth pitch, Opus 5.5's and Fable 5.1's at an account limit and Sonnet 5.5's by its own choice; partial attempts get no score.
They touch few pitches, all foul, and never a curveball or changeup
Robot control
The four complete attempts touched 4 of their 40 pitches, all foul, so none earned carry; GPT‑6.1 Terra touched none of its ten.
Of the 5 contacts across all 7 attempts, 4 came on sliders and 1 on a fastball; no curveball or changeup was touched.
GPT‑6 Luna's one contact carried 79 m and GPT‑6 Astra's 30 m, but both went foul and earned a point each.
Sonnet 5.5 ended its own attempt after two pitches, both missed, though its own output said the session was still live.
What an attempt takes, and what the environment includes
An attempt
The four complete attempts ran about 21 minutes each and sent 21 to 58 moves, typically 5 to 8 seconds after the image.
What's in it
The batting simulator and renderer with their tests, the tool server, the brief, the score checker, and all seven attempts' records, replays and videos.
Format
Runs on Node.js, with headless Chromium for the camera and Python for the tools; today on a Windows PC, with other packaging on request.
Fresh variants
Other pitch sequences in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials