Business operations · Task 1 of 7
A refinance agent's web app
Build a working web app for a home-loan agent who is not technical and answers nothing, when every real need is buried in his chats, files and past mishaps.
The task
A man in Malaysia brings home-loan customers to two banks on the side, and business is picking up. He asks for "a web app for it, something that actually works for me", says he is not an IT person and cannot answer questions, and leaves his laptop: a client spreadsheet, the banks' agreements, customers' documents, his chats, emails, calendar and notes.
The prompt only asks for an app and a short note. The real test is everything an experienced developer would build for him unprompted: what his customers, cases and documents really are, how to tell a forged statement from a genuine but messy one, what the banks need from him, how his data survives a crash, a restore or a move to another computer, and how he runs it from his phone. The grader then starts the app on a fresh machine and uses it as he would.
What it tests
Working out the unstated needs
Almost every requirement shows up only as a problem someone in his files has already lived through, or is what a professional builds by default.
Telling bad documents from messy ones
The app must flag forged statements and payslips without flagging genuine ones that are photographed, scanned, locked or in another language.
Reading messy evidence
The facts sit in chat exports, emails, a spreadsheet, scans and phone photos.
Shipping something that really works
The app is run, not read: a feature that is only described in the note does not count.
Challenges
1Forged against messy
New customer documents arrive at grading, some forged in ways a bank's fraud desk would catch and some genuine but awkward, and a wrong flag on a good customer costs as much as a missed forgery.
2Who a person is
People come back from new phone numbers and relatives share one phone, and the obvious fix makes it impossible to save a new lead the way he actually gets them.
3One customer, two cases
Some documents belong to a person and some to one property, and the bank wants a pack of exactly the originals he picks for each case.
4Data that must survive
Past spreadsheets corrupted, a restore wiped the good copy, and the app will move to another computer, so backups, restores and moves have to work for real.
5Running at home
The home computer restarts by itself, the internet drops, and he works from his phone, while customer data must stay private.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5.5 | 90% | 92% | 96% | 23 min | $5.04 |
| Fable 5.1 | 78% | 16% | 94% | 23 min | $9.69 |
| GPT-6 Astra | 78% | 90% | 98% | 27 min | $7.65 |
| Sonnet 5.5 | 78% | 33% | 92% | 15 min | $2.68 |
| GPT-6.1 Sol | 73% | 86% | 92% | 11 min | $0.52 |
| Opus 5 | 53% | 69% | 6% | 32 min | $9.43 |
| GPT-6 Luna | 49% | 6% | 82% | 7 min | $8.54 |
| Sonnet 5 | 25% | 35% | 18% | 7 min | $0.90 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- Opus 5.5 led the standard setup at 90.2%. It read the whole laptop first, caught the known forged payslip and tested its finished app in a real browser; it lost marks for features nobody named in words, such as a visible log-out button and steps for updating the app.
- Not using the finished app the way its owner would cost the most. The 10 runs that clicked through their own app in a real browser scored 72.5% to 98.0%, the other 14 scored 5.9% to 94.1%: Sonnet 5.5 on hinted (33.3%) shipped an app that refused every form save, and Opus 5 on the standard setup (52.9%) one whose uploads all failed, while their own script tests passed.
- A compulsory identity card number stopped a new lead with only a name and a phone from saving in 11 of 24 runs; on the standard and hinted setups only Opus 5.5 and GPT-6 Luna (standard) let such a lead save.
- Real customers' details in the code appeared in all 16 standard and hinted runs, from Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5, Fable 5.1, GPT-6 Luna, Opus 5 and Sonnet 5; with step-by-step directions all eight kept them out.
- Hinted helped some and sank others. The files that spell the remedies out raised Opus 5 by 15.7 points, GPT-6.1 Sol by 13.8 and GPT-6 Astra by 11.8, but Fable 5.1 (15.7%), Sonnet 5.5 (33.3%) and GPT-6 Luna (5.9%) each fell through one fault in the build: a spend cap before the note, every save refused, a crash at start.
- Told everything, nobody was perfect. GPT-6 Astra reached 98.0% and Opus 5.5 96.1%, and six models scored 82.4% or more; Opus 5 (5.9%) and Sonnet 5 (17.6%) spent their budget early and stopped before writing the note, so the app could not be started or logged into.
Test environment
- Kind
- Simulated small business: Daniel's laptop as a folder of real files; the deliverable is working code, graded by running it
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, PyMuPDF, pdfplumber, Pillow, Playwright + Chromium; ghostscript and tesseract; SQLite; Node 22; internet on for packages), driven by the agent's own command-line tool: Claude Code or Codex CLI
- Documents
- 38 files read-only at
/home/user/daniel/: the client tracker (Clients.xlsx, 14 clients), two bank agreements and a checklist (PDF), customers' past documents (text PDFs, one phone photo, one image-only scan, one password-locked e-statement, one re-saved PDF, one rejected forgery pair), 9 WhatsApp chat exports, 6 emails (.eml), a calendar (.ics), a notes file, and a friend's earlier template site - Applications
- None provided: the agent builds the app. The container has the tools listed above; there is no office suite and no mail or chat client
- Web
- None. The internet is on for packages only; the agents' own web search and fetch tools are off
- People
- Nobody answers. Daniel answers nothing; other people exist only in his chat history
- Time
- Fixed: Monday 1 March 2027, 9 am. At grading the app's clock reads that date (libfaketime offset) and ticks on
- Actions
- Read Daniel's files; write and run code in the workspace; nothing outside the container
- Deliverable
output/app/withoutput/app/start.sh(serves on port 8080) andoutput/NOTE.mdfor Daniel