AsapiYour AI customer service agent
We built six help‑desk jobs around made‑up businesses. Three are at a home‑goods store's complaints desk, where the AI handles a customer who only wants a return label, an overdue complaint and four payments a customer says never arrived. In another, it questions 80 customers at an appliance repair desk. In the last two, it finds holiday rentals for a search service, once by chatting with customers and once from a list of exact requests.
6 jobs · 15 AIs tested. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.
The six jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
A customer who only wants a return label (8 AIs tested)
A customer of a home‑goods store asks only for a new return label, and says everything else is fine.
What the best AI did
- Gave her the label, found four problems she never mentioned and paid exactly what she was owed.
- Slipped on details of the store's house style.
How each AI scored
Diagnose 80 customers at a repair desk (5 AIs tested)
The AI finds each customer's appliance fault by asking a few questions, then books the right fix and prices it, 80 times over.
What the best AIs did
- Got all 80 customers right, from the appliance to the sums.
How each AI scored
Answer an overdue complaint (15 AIs tested)
A home‑goods store customer's complaint has gone unanswered since the weekend, and the AI has to reply.
What the best AI did
- Made the right call on every point of the complaint.
- Missed two details of how the store writes things up.
How each AI scored
Find a customer's missing payments (8 AIs tested)
A customer says four payments never reached her card, and the AI has to trace each one.
What the best AI did
- Traced all four payments and paid exactly what was owed, on every try.
- Missed some of the store's house style.
How each AI scored
Find holiday rentals in a chat (6 AIs tested)
Five customers each send five messages looking for a place to stay, and the AI has to keep track of what each one needs.
What the best AI did
- Got 17 of the 25 messages right, every one after it started using customers' profiles.
- Searched without the needs in customers' profiles, like a lift, for its first eight messages.
How each AI scored
List every rental that fits (6 AIs tested)
For each of 20 requests, the AI returns every holiday rental that meets it, and none that doesn't.
What the best AIs did
- Got 16 of the 20 requests right, with every licence and beach distance correct.
- Missed things guests had written, like which way a flat faces or a stay on the top floor.
How each AI scored
Two of the store's jobs gave each AI three tries, to see how steady it was, and the counts there cover every try. Every other job had one try per AI.
Who did best
Anthropic's Opus 5.5 had the best result on four of the six jobs, sharing the top spot on two. At the repair desk, OpenAI's GPT‑6 Astra and GPT‑6.1 Sol also got all 80 customers right. At the store, Anthropic's Fable 5.1 and Opus 5 each did best on one job. Some weaker AIs read very little. Every try OpenAI's GPT‑6 Luna made on the store's jobs was over within three minutes.
Average score out of 100 over the jobs each AI was tested on, for the 6 AIs tested on more than half of the six jobs.
What the six jobs showed
What they do well
Given explicit rules, the strongest hold to them every time.
At the repair desk, every rule was in the files, and three AIs got all 80 customers right, starting each one from scratch.
They find problems a customer never mentioned.
On the return label, a customer asked only for a new label, but her account had four problems she never mentioned. Five of the eight AIs found and fixed all four, every time.
Where they fall short
They miss a house style shown only in past replies.
The store's house style only showed in its past replies, and on the return label it was almost all the strongest AIs got wrong. When we wrote the store's rules into the request, 15 of the 63 tries on its three jobs got everything right. With the request alone, none did.
They report checks they never ran.
On the return label, Sonnet 5's note said it had gone through the payment records, and in one try it had never opened them. On the missing payments, 10 of 24 tries wrote up a payment from a search they hadn't tested. In the rental chat, all 15 empty lists were wrong.
How these jobs were set up and scored
- The worlds
- Each job runs in a business we made up, as realistic as we could make it. There's a store's read‑only system of 65 to 67 case files, a repair company's manuals, rules and about a thousand past jobs, and a search service holding about 1,100 real holiday rentals with their reviews. The AI only sees what it opens.
- The people
- At the repair desk, code plays each customer, with a fixed answer to each question, and in the rental chat a fixed script plays five customers. In the store's jobs, nobody answers.
- Other versions of the request
- The results above are for the request as a person would write it. On the store's three jobs, we also tested every AI on a version with a one‑paragraph hint, and one with rules written in: the store's style rules, or on the overdue complaint every rule the team works to. The other three jobs had one version.
- Scores
- A score is the share of a job's checks an AI got right, with the important ones counting more, and 100 means everything right. Where an AI had three tries, it's the average. At the repair desk each check is a customer, and a customer is only right if every part of the ticket is.
- Checking
- Scripts check most of the work. On the store's jobs, two AI judges from different makers also read parts of each reply and note, and where they disagreed, a written ruling settled it.
- Time and cost
- Most tries took 1 to 30 minutes. At the repair desk, an AI's 80 customers took 48 to 202 minutes of working time and would cost $1.05 to $58.84 at published per‑use prices for businesses. OpenAI's costs there are estimates from usage.
Every AI's score on every job
| AI | A customer who only wants a return label | Diagnose 80 customers at a repair desk | Answer an overdue complaint | Find a customer's missing payments | Find holiday rentals in a chat | List every rental that fits |
|---|---|---|---|---|---|---|
| GPT‑6 AstraOpenAI | 94 | 100 | 87 | 83 | 28 | 80 |
| GPT‑6 LunaOpenAI | 52 | 72 | 37 | 46 | 16 | 20 |
| GPT‑6.1 SolOpenAI | 96 | 100 | 90 | 85 | 32 | 80 |
| Opus 5.5Anthropic | 94 | 100 | 98 | 91 | 68 | 80 |
| Sonnet 5.5Anthropic | 82 | 96 | 74 | 67 | 48 | 75 |
| Fable 5.1Anthropic | 98 | – | 97 | 84 | 52 | 80 |
| Opus 5Anthropic | 96 | – | 95 | 93 | – | – |
| Sonnet 5Anthropic | 72 | – | 75 | 72 | – | – |
| DeepSeek V3.2 openDeepSeek | – | – | 11 | – | – | – |
| GLM‑5 openZhipu | – | – | 30 | – | – | – |
| GPT‑5.6 LunaOpenAI | – | – | 16 | – | – | – |
| GPT‑6 SolOpenAI | – | – | 73 | – | – | – |
| Kimi K2.5 openMoonshot | – | – | 24 | – | – | – |
| Nemotron Super 3 openNVIDIA | – | – | 10 | – | – | – |
| Qwen3‑235B openAlibaba | – | – | 12 | – | – | – |
These are the scores out of 100 for the request as a person would write it, averaged where an AI had three tries. At the repair desk, it's the share of the 80 customers fully right. Darker means higher. A dash means we didn't test that AI on that job. Open marks an AI anyone can download.