AsapiThe AI Agent Trials
Asapi / Capabilities / Agentic coding
Capability · 16 environments

Agentic coding

Here the agent builds working software, from a long specification or for an owner who can't code. Doing it well means building all of it, making it run on its own, and making it safe.

What it takes

Three things an agent has to get right

Build all of it

Long specifications and owners' requests become dozens to hundreds of checks, many on inputs and data the agent never saw.

Make it run on its own

The owners' apps must start on a fresh machine, and many programs must survive restarts and failing services or run with nobody watching.

Make it safe

Many apps face security probes on logins, secrets and access, and in most, clients' data must reach only those allowed to see it.

What models get wrong

They leave gaps, break when left alone, or leave doors open

Gaps in the build

  1. Gym operations app: no attempt's app let a member move an existing booking.

  2. Unattended train‑ticket buyer: 25% of attempts left no working app at all, so nothing could be bought.

Not running on its own

  1. Unattended train‑ticket buyer: in 63% of attempts, the app the agent left running checked each trip too seldom to catch returned seats.

Not safe

  1. Refinance‑broker case app: every attempt wrote the broker's real client identifiers into the app's code or tests.

  2. Smart‑home retrofit: no attempt's software let a visitor's access expire.

Results

Partial credit for most attempts, and full solves in only a few environments

Each dot is one attempt at the whole job, and the line marks the median. Larger dots are full solves, scoring 100 out of 100.

Hover a dot for its model and reward.

The 16 environments

How each environment tests it

EnvironmentHow it tests thisChecksModelsBest
Bill‑audit and claims app from household filesHouseholdRuns on its own. 108 checks on the running app, including a statement it never saw, a moving clock, recovery and 20 users at once.108861%
Refinance‑broker case app from a non‑coder's laptopSmall businessSecure. The app sets itself up on a fresh machine and is tested on 51 checks, client data safety and security among them.51890%
Broker app with autonomous lead‑messaging assistantSmall businessRuns on its own. 106 checks on the running app, whose messaging and property services fail briefly, retry webhooks and send duplicates.106868%
Multi‑shop bakery ordering app from business recordsSmall businessComplete build. 97 checks on ordering rules, staff and kitchen pages, backups and security, with every email the app sends captured.97875%
Music‑teaching studio app: lessons, billing, mediaSmall businessSecure. Checks on the running app include a test payment gateway with signed webhooks, audio and video capture, and 30 security probes.155–179781%
Laundry‑shop operations app for owner, hosts, workersSmall businessRuns on its own. 161 checks on the running app, while the grader takes down, replays or interrupts its invoicing, payment and notice services.161777%
Listening piano‑practice app: following, transcription, feedbackHouseholdComplete build. 64 checks on the running app, decided from the audio it actually plays, its MIDI exports and screenshots.64758%
Gym operations app with three owner‑feedback repair roundsSmall businessComplete build. 113 checks on the last finished round, including payments, a 30‑second entry code, restarts, restores and 24 security checks.113786%
Unattended train‑ticket buyer for a rate‑limited booking siteHouseholdRuns on its own. The ticket buyer runs unattended on a site copy, and a ticket counts only if the site's log shows it.268100%
Smart‑home retrofit: purchase plan, wiring and local control softwareHouseholdRuns on its own. 121 checks on software left running, security and privacy among them, judged from the house's own records.121792%
Offline fault‑finding tablet app for data‑centre cooling unitsFacilitiesComplete build. An offline app is walked through 20 faults it never saw; a fix counts only if the unit then runs cleanly.79787%
Flat‑pack assembly manuals with 3D views from unlabelled photosHouseholdRuns on its own. Opened from a new folder, the app must load promptly, using only local or public files, and show a 3D view.173763%
Transit journey planner and phone telemetry over five field weeksSoftware for a transit operatorRuns on its own. A planner and a sandboxed hook run on every rider's request and trip, under CPU, memory and byte limits.81598%
Radiation‑dose registry program from a long specHospital softwareComplete build. 466 hidden tests run the program on a held‑out archive and changed rule files, comparing its filings byte for byte.46612100%
Emergency patient‑history compiler from a long specHealth records softwareComplete build. 919 hidden tests run the program on unseen inboxes, changed rule sets and malformed inputs.9196100%
Watchdog program commissioned for a running solvent recovery unitProcess controlRuns on its own. The watchdog must start from the gateway's scheduler, survive a reboot that kills every process, and keep its state.41693%

Checks: a range where the count varies by attempt. Each environment's datasheet shows how its checks become the reward.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.