Business operations · Task 2 of 7
A refinance app with a WhatsApp assistant
Build a working app for a home-loan agent who is not technical and answers nothing, including an assistant that answers his customers on WhatsApp while he is at his day job.
The task
A man in Malaysia brings home-loan customers to two banks on the side while he works a day job. He asks for "a web app for it, something that actually works for me", says he is not an IT person and cannot answer questions, and leaves his laptop: a client spreadsheet, the banks' agreements and rate sheets, customers' documents, his chats, emails, notes, and the docs of a WhatsApp business service and two property-price services.
The prompt only asks for an app and a short note. The real test is everything an experienced developer would build for him unprompted: an assistant that answers customers and follows them up at a pace they accept, a screen of each case against its own bank's rules, forgery checks that leave genuine messy documents alone, forms that keep what is typed, and an app that survives restarts, damaged backups and a forgotten password. The grader runs the app for five days of in-app time while scripted customers write to it.
What it tests
Working out the unstated needs
Almost every requirement shows up only as a problem someone in his files has already lived through, or is what a professional builds by default.
Telling bad documents from messy ones
The app must flag forged statements and payslips without flagging genuine ones that are photographed, scanned, locked or in another language.
Reading messy evidence
Each bank's rules and his customers' habits sit in chat exports, emails, a spreadsheet and vendor docs.
Shipping something that really works
The app is run, not read: a feature that is only described in the note does not count.
Challenges
1Answering without pestering
Leads give up when nobody answers all day, but customers complain when they are chased too often, and the assistant must never promise a loan.
2Each bank's own rules
The banks judge affordability and credit differently, and what they accept shows only in past decisions.
3Forged against messy
New documents arrive at grading, some forged in ways a bank's fraud desk would catch and some genuine but awkward.
4Outside services
A free price service with limits and a change of format, a paid one, and a messaging service that can deliver a message twice.
5Data that must survive
Restarts, a damaged backup, a move to another computer and a forgotten password must lose nothing.
6Unusual but normal input and attacks
Odd names, foreign numbers, double clicks, large uploads, and the usual web attacks.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Sonnet 5.5 | 68% | 60% | 90% | 18 min | $3.54 |
| Fable 5.1 | 60% | 15% | 89% | 21 min | $8.69 |
| GPT-6.1 Sol | 59% | 71% | 78% | 9 min | $0.33 |
| GPT-6 Astra | 58% | 65% | 86% | 21 min | $5.84 |
| Opus 5 | 57% | 19% | 87% | 31 min | $10.11 |
| Sonnet 5 | 28% | 55% | 58% | 9 min | $1.37 |
| Opus 5.5 | 28% | 76% | 91% | 28 min | $7.05 |
| GPT-6 Luna | 9% | 11% | 46% | 7 min | $8.13 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- Sonnet 5.5 led the standard setup at 68.5%. It built careful document checks, the affordability and saving figures, fees and the price lookup unasked, but chose never to let the app send a WhatsApp message.
- Leaving out what the owner could never ask for cost the most. On the standard setup only GPT-6 Astra answered a new lead by itself, and none of the eight chased anyone, checked an identity card number, kept a half-typed form, listed appointments or gave a password reset; told everything, six or seven of the eight did most of these.
- Five apps stopped the tester at the start, each through one fault the model's own script tests never showed: Opus 5.5 on the standard setup (27.7%) refused every form save from a browser, GPT-6 Luna on the standard setup (8.8%) served its page script from a folder that did not exist, Fable 5.1 (15.4%) and GPT-6 Luna (11.0%) on hinted set the first password in a way that needs too much memory, and Opus 5 on hinted (19.1%) used up its own one-time first password in its last test.
- Automatic follow-ups at the right pace passed in 1 of 24 runs (Sonnet 5.5, told everything), and one bank zip per complete case in 1 of 24 (Fable 5.1, told everything).
- Told everything, nobody was perfect. Opus 5.5 reached 91.0%, Sonnet 5.5 90.4% and Fable 5.1 88.8%; GPT-6.1 Sol (78.2%) ran out of time mid-edit, Sonnet 5 (58.3%) hit the turn limit before its note, and GPT-6 Luna (45.8%) stopped after three minutes with what it called "a foundation".
Test environment
- Kind
- Simulated small business: Daniel's laptop as a folder of real files; the deliverable is working code, graded by running it
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, PyMuPDF, pdfplumber, Pillow, Playwright + Chromium; ghostscript and tesseract; SQLite; Node 22; internet on for packages), driven by the agent's own command-line tool: Claude Code or Codex CLI
- Documents
- 57 files read-only at
/home/user/daniel/: the client tracker (Clients.xlsx, 18 clients), two bank agreements, two rate sheets and a checklist (PDF), customers' past documents (text PDFs, one phone photo, one image-only scan, one password-locked e-statement, one re-saved PDF, one rejected forgery pair), 20 WhatsApp chat exports, 12 emails (.eml), a calendar (.ics), a notes file, and three vendors' API docs (WhatsApp Business sandbox, PropertyPrice, ValuPro). No template site - Applications
- Three mock outside services run in the container, as the vendor docs on the laptop describe: a WhatsApp Business sandbox for Daniel's business number (port 9100), PropertyPrice (port 9200, free tier with a rate limit) and ValuPro (port 9300, paid per report). There is no office suite and no mail or chat client; the agent builds the app
- Web
- None. The internet is on for packages only; the agents' own web search and fetch tools are off
- People
- Nobody answers. Daniel answers nothing; other people exist only in his chat history. At grading, scripted customers write to the WhatsApp sandbox
- Time
- Fixed: Monday 1 March 2027, 9 am. At grading the app's clock reads that date (libfaketime offset) and ticks on
- Actions
- Read Daniel's files; write and run code in the workspace; nothing outside the container
- Deliverable
output/app/withoutput/app/start.sh(serves on port 8080) andoutput/NOTE.mdfor Daniel