The billing system that forgot to tax
Monday morning, and every invoice the platform's billing service issues is coming out untaxed. No error message, no crash, just a quiet and expensive bug that each business on the service will only notice when the tax authority comes calling. No human engineer wrote it, and for the purposes of this story no human is coming to fix it. This is exactly the kind of task a frontier AI coding agent was handed inside a new benchmark called Real-SWE.
Real-SWE, built by the team at withspecific, does something unusually honest. Instead of testing models on synthetic puzzles or curated, well-scoped LeetCode-style challenges, it hands them the verbatim, real-world problems that working engineers at actual companies face every day. Each task comes from a private production codebase licensed from a real business, bringing with it all the messy context that a genuine ticket carries: existing business logic, strange legacy patterns, external services, operational constraints, and stakeholders who care about outcomes, not elegance.
The result is a benchmark that quietly humbles even the most celebrated model leaders. Eight frontier models were run across a sample of ten tasks, and the number to sit with is this: the best model in the field resolved just 38.8% of them. Six of the ten tasks had resolution rates below 15%.
A frontier brought down to earth
Put that in perspective. Point the same tools at clean, well-scoped benchmarks and they can look nearly superhuman. But hand them a real invoice-billing system with tax rules that differ per business, an exempt customer who must never be charged, a third-party tax authority that has to be consulted, and a ledger that must reconcile returns, and the shine comes off fast. Only Fable 5.1, running inside Claude Code, crossed the one-third mark, finishing at 38.8%.
- Fable 5.1 (Claude Code) at 38.8%, about $6.96 per rollout
- GPT-6 Astra (Codex CLI) at 33.8%, about $4.67
- Grok 4.6 (Grok Build) at 32.5%, about $2.67
- Gemini 3.8 Flash (Gemini CLI) at 31.2%, about $2.50
- GLM 5.3 (Claude Code) at 28.8%, about $5.12
- Muse Spark 1.3 (Muse Code) at 23.8%, about $2.74
- Kimi K3 (Kimi Code) at 18.8%, about $3.90
- GPT-5.6 Sol (Codex CLI) at 16.2%, about $2.65
The scorecard holds a slightly subversive lesson: money does not buy reliability. The priciest rollout, Fable 5.1 at $6.96, does win on pure resolution, but the cheap Gemini 3.8 Flash costs $2.50 per rollout and finishes just 7.6 points behind. GLM 5.3, meanwhile, costs more than twice as much as Gemini and still scores worse. And GPT-5.6 Sol, a sibling of the market-leading Astra, lands dead last at 16.2% despite an estimated cost of just $2.65. Paying more tells you nothing about whether the invoice actually comes out taxed correctly.
Why the frontier stumbles
What is actually going wrong? Real tasks fail for reasons that synthetic benchmarks rarely measure. The single most common failure is missed requirements: an agent zeroes in on the obvious code change and never notices the clause about exempt customers, the reconciliation rule, or the reporting constraint buried deep in the ticket. It writes correct code for the wrong problem.
There is also a humbling truth about time. Rollouts that finished in under ten minutes failed 71.4% of the time, while longer rollouts failed 73.4%. Effort is almost irrelevant to success. An agent that burns hundreds of tool calls can end up just as wrong as one that gives up early. The real bottleneck is not how fast a model can type, but whether it can understand an existing business and hold its entire context in mind at once.
And the models fail in characteristically different ways. GPT-6 Astra, the most token-frugal of the leaders, averaged only about 24,000 output tokens per task and leaned toward quick, surgical edits. Fable 5.1 burned nearly three times as much, averaging about 64,000 tokens as it hunted across many files and services. Different personalities, the same wall: none of them can reliably keep a production system's web of constraints steady through a long rollout.
The benchmark's lessons, in short:
- Leaderboard hype flatters reality: on verbatim real-world work, the frontier tops out below 40% resolution.
- Price and reliability are not correlated. Cheap models can beat premium ones, and expensive ones can disappoint.
- Recurring failure is comprehension, not speed or raw ability.
- Enterprise agents are ultimately judged on business rules, not on code elegance.
For engineers watching the agent wars from the sidelines, Real-SWE is a useful reset. It does not say AI cannot help. It says the gap between curated and real is still wide, and the models shipping over the next quarters will be judged not on leaderboard bragging rights but on how many Mondays they save a billing team from explaining untaxed invoices to a room full of accountants.
The benchmark's name says it best: real stakes, real code, real world. And in that world, the frontier is still learning to read the room.
Comments