Baseball batting by a bimanual robot from batter‑eye images

Ten pitches, a world that never pauses, and only a camera image

Role

The controller of a simulated two‑handed robot holding a bat, acting through four tools; no script, controller program or scheduled moves are allowed.

Request

Look first, then start the ten pitches; track each from the camera image, swing by setting shoulder and elbow targets, then bring the bat back for the next. Aim for the highest total score, then finish.

Deliverable

The joint commands and the engine's record of each pitch: contact or miss, fair or foul, and how far the ball carried.

What the reward depends on

Contact on each pitch, and directing it fair: a fair hit earns a point a metre of carry on top of the contact point, so one long fair hit outweighs many fouls.

Where the requirements are

Stated in the brief. The ball's path shows only in the images, and the bat's actual pose only in how it looks; pitch types and order aren't given.

A continuous physics world, seen through one camera

Interface

Four MCP tools (look, start, move and finish) through Codex CLI for the OpenAI models and Claude Code for the Anthropic ones, at low reasoning effort, with the CLIs' shell and web tools off.

Observation

One 960 by 540 colour image from the batter's eyes per look or move, with the public score and timing; no ball coordinates, spin or joint readings.

Control

Eight joint targets, shoulder yaw, pitch and roll and the elbow on each side, driving position servos with speed and torque limits. Targets hold until replaced.

The bat

Both hands grip the bat rigidly and the wrists swivel freely, so the bat lies along the line between the hands and both sides must move together. No swing, aim or return command exists.

Physics

Fixed 1 ms steps, with drag and an approximate spin force on the ball, running continuously at a 50th of real speed, through the agent's thinking and tool calls; nothing pauses.

Pitches

Ten in a seeded shuffled order, the same for every model: three fastballs, three sliders, two changeups and two curveballs, with varied speed, spin and placement; types aren't shown.

Limits

One session: the ten pitches take about 21 minutes of wall time, and the runner stops a model at 40 minutes.

Isolation

CLI tool restrictions and, for Codex, a hook guard, not an operating‑system container. The seven attempts overlapped on one PC.

Points per pitch: 1 for contact, plus 1 a metre of fair carry

Structure

Ten pitches: a miss earns 0, any contact 1, and a fair hit 1 more per metre of horizontal carry to its first bounce. A foul earns only the contact point; there's no maximum.

Who decides

The engine records each pitch's contact, direction and carry; a script checks that the pitches add up to the engine's total and that all ten were seen. No model judge.

Cut‑short attempts

An attempt that stops before the tenth pitch gets partial points but no score, and isn't ranked; the pitches go on with the targets held, and that play is excluded.

Replay

Each attempt's recorded actions replay deterministically to the same final state; this was checked for all seven.

On these pages

Raw points have no ceiling, so 1,000 points show as 100%; on that scale every attempt so far rounds to 0%.

Formula

Score = contacts + metres of fair carry, summed over the ten pitches.

The four complete attempts made four contacts, every one foul

Of the four attempts that saw all ten pitches, GPT‑6.1 Sol made 2 contacts, GPT‑6 Astra and GPT‑6 Luna 1 each and GPT‑6.1 Terra none, all foul, so no carry counted. The median complete attempt scored 1 point; on our scale, all round to 0%.

Each dot is one attempt; the line marks the median.

One attempt per model, seven models, on the same seeded pitches. The chart and table show the four that saw all ten pitches.

Three attempts were cut short and are unranked: Opus 5.5 and Fable 5.1 by their subscription's session limit, after seven and six pitches with 0 and 1 points, and Sonnet 5.5, which ended its own session after two.

All seven were made again on 10 November 2026 after a correction to the joint dynamics; earlier results are set aside.

What the models got wrong

Counted over the 4 attempts that saw all ten pitches, unless a line says otherwise.

1. No fair contact. The 4 complete attempts touched 4 of their 40 pitches, all foul, so none earned carry; GPT‑6.1 Terra touched none of its ten.

2. Curveballs and changeups untouched. Of the 5 contacts across all 7 attempts, 4 came on sliders and 1 on a fastball; no curveball or changeup was touched.

3. Distance without direction. GPT‑6 Luna's one contact carried 79 m and GPT‑6 Astra's 30 m, but both went foul and earned 1 point each.

4. Stopping while live. Sonnet 5.5 ended its own attempt after two pitches, both missed, though it said the session was still live.

What an attempt costs

Length

Each session ran about 21 minutes of wall time, fixed by the pitches. The complete attempts sent 21 to 58 move commands; their median lag from the image seen to the move arriving was 4.9 to 7.9 seconds.

Cost

Not recorded: every attempt ran on a subscription sign‑in, which logs no cost or provider requests, so we give no figure.

What you license

With a licence

Revision 2: the batting simulator and renderer with their tests, the tool server and launchers, the brief, the score checker, and each of the 7 attempts' recorded actions, outcomes and replays, with review videos, as baseline results.

Format

Runs on Node.js, with headless Chromium rendering the camera and Python for the tool server and launchers; today on a Windows PC, outside a container. Other packaging on request.

Fresh variants

Other pitch sequences in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model, on one seed, so we haven't measured how much attempts vary.
• No reference: no permitted demonstration of long fair hits under image‑only control exists yet, so the low scores can't separate model difficulty from interface difficulty.
• Three of the seven attempts were cut short, two by a subscription's session limit, so only four are complete and ranked.
• Latency is part of the job, and the seven attempts shared one PC, with camera images captured in two queues of four and three models, so their timings include contention.
• Simplified physics: approximate ball aerodynamics, joint servos and grip, so results don't speak to real robots.
• Raw points have no ceiling, so our 0 to 100% scale compresses every attempt so far to 0%.