What happened
Specific Labs, a two-year-old startup whose business is licensing operational data and codebases from real companies, published a benchmark called Real-SWE on September 12, 2026. Unlike most public coding benchmarks, which run on open source repositories that models have already seen during training, Real-SWE's tasks come from private, production codebases the company licensed directly from real businesses: an events app with more than 200,000 users, a consumer fintech platform that has processed more than 100,000 bank statements, and enterprise sales software. The tasks are not toy problems. One published example asks an agent to fix invoice billing so each business charges the correct tax, exempt customers are not taxed, and the fix stays consistent across a tax provider, a production ledger, and existing invoices. The published results cover eight model-and-coding-tool combinations run on ten disclosed sample tasks, each attempted eight times, for 640 scored attempts in total. Anthropic's Fable 5.1, running inside Claude Code, topped the leaderboard by resolving 38.8% of tasks. OpenAI's GPT-6 Astra, in its own Codex CLI tool, followed at 33.8%, then Google's Gemini 3.8 Flash in Gemini CLI at 31.2%, Z.ai's GLM 5.3 in Claude Code at 28.8%, xAI's Grok 4.6 and Meta's Muse Spark 1.3 tied at 23.8%, Moonshot's Kimi K3 at 18.8%, and OpenAI's own GPT-5.6 Sol last at 16.2%. On six of the ten disclosed tasks, every single model and tool tested solved fewer than 15% of attempts, and one task was solved zero times across all 64 attempts run against it.
Why it matters for business owners
Any business already running an AI coding agent, whether through an internal engineering team, a development agency, or a vendor's own tooling, is relying on a version of the promise this benchmark tests: that a coding agent trained mostly on public code can reliably do useful, correct work on code it has never seen before, inside a system with its own conventions, dependencies, and business rules. The public benchmark scores most vendors publish, and the demos most sales teams show, are typically run on code closer to what a model has already studied. Real-SWE is one of the first widely discussed, independently reproduced attempts to measure the gap between that marketing number and a coding agent's performance on an actual, unfamiliar private codebase. The result that matters most for a buying decision is not the ranking. It is the cost data Specific Labs published alongside it. Divide each model's published cost per attempt by its resolution rate, and the ranking changes: Gemini 3.8 Flash, the third-place model by accuracy, was the cheapest way to get one working change, at roughly $8 per resolved task. Fable 5.1, the top-ranked model, cost roughly $18 per resolved task, more than double, because a coding agent's failed attempts are not free. Someone still has to notice the failure, and in most business settings that someone is a paid engineer.
What owners should not misunderstand
This is not proof that any specific vendor's coding agent is bad, or that AI coding agents are not worth using. The disclosed sample is ten tasks. The gap between the top three models sits close to what an independent analysis of the same data called noise, meaning it is not a reliable basis for ranking those three against each other specifically. The underlying private codebases cannot be inspected by outside researchers, so the results rest on trusting Specific Labs' own screening and scoring process, even though two independent outlets reproduced the same published numbers from the source. It is also worth saying plainly: Specific Labs' business is licensing private company data and code to AI labs for training. A benchmark showing frontier coding agents struggle on private, unfamiliar code is also a sales argument for that business. That does not make the published numbers wrong, but it is a reason to treat this benchmark the same way this blog has argued a business should treat any vendor's own capability claim: as a data point to verify against your own situation, not a verdict to act on directly. Finally, each model was tested inside its own maker's coding tool (Fable 5.1 in Claude Code, GPT-6 Astra in Codex CLI, and so on), so the benchmark measures a model-and-tool combination, not a model in isolation. It cannot tell a business how a given model would perform inside a different coding tool, which matters if your team already has a preferred development environment.
The operational lesson
The standard advice for choosing software has always been to check the vendor's own performance claims against your actual use case before buying. Real-SWE is a concrete illustration of why that advice matters even more for AI coding agents than for most software: the same model can look strong on a public leaderboard and resolve fewer than one attempt in six on a business's own kind of code, and the model that resolves the most tasks is not automatically the cheapest way to get one working change. Whether the higher-accuracy, higher-cost model or the lower-accuracy, lower-cost model is the better economic choice depends on one thing this benchmark cannot answer for you: how expensive it is for your business to catch a bad attempt before it ships. A team with a strong automated test suite that reliably rejects a wrong change can afford a cheaper, less accurate agent, because bad attempts get caught for free. A team relying mostly on a person reading the code before it merges pays a real cost for every extra failed attempt, and a pricier, more accurate agent can work out cheaper overall once that review time is counted. The benchmark's failure data adds a second, practical point: the models did not fail in the same way. One model's failures were mostly missed requirements. Another's were mostly integration errors, where the idea was right but the change was wired into the surrounding system incorrectly. A third model's failures were mostly unverified assumptions about how the existing system worked. A single generic code-review checklist will not catch what your specific coding agent actually tends to get wrong. The review process should match the tool.
What a serious business should do next
Do not choose or switch an AI coding agent based on a public leaderboard rank, a vendor demo, or a marketing screenshot alone, including the leaderboard in this benchmark. Do run a small pilot using real tasks from your own codebase, the kind of work your engineers actually do, before committing budget or expanding an existing coding agent's access. A handful of representative tasks will tell you more about fit than any public score. Do track cost per resolved task once you have real usage, not just the subscription price or the advertised per-token rate. A cheaper-looking agent that fails more often can cost more once you count the time spent catching its mistakes, and a pricier agent can be worth it if your review process is thin. Do build (or check) a review process suited to the specific agent you use, since different coding agents tend to fail in different ways. A generic checklist misses what your tool actually gets wrong most often. Do not treat this benchmark, or any single benchmark, as a final verdict. Notice who published it, what they sell, and how small the sample is, the same discipline this blog has recommended for any vendor's own capability claim.
The Atlacis view
The interesting part of this story is not which AI lab won a coding leaderboard this week. It is that even a well-built, independently reproduced benchmark cannot tell a specific business what a coding agent will actually cost it, because that number depends on your codebase, your review process, and how expensive a mistake is for you to catch. Atlacis helps business owners slow down before an AI tool purchase gets made on a leaderboard screenshot or a vendor demo, run the small test that actually matters for their own workflow, and understand the real cost of a tool once failed attempts and review time are counted, before signing up for a plan that does not fit how the business actually works.
The short version
- On September 12, 2026, Specific Labs published Real-SWE, a benchmark testing AI coding agents on licensed, private production code rather than public GitHub repositories.
- The top-ranked model and tool combination, Anthropic's Fable 5.1 in Claude Code, resolved 38.8% of tasks; on six of the ten disclosed tasks, every model tested resolved fewer than 15% of attempts.
- Cost per resolved task inverted part of the leaderboard: Gemini 3.8 Flash was the cheapest way to get one working change (about $8), while the top-ranked model, Fable 5.1, cost more than double (about $18), once failed attempts are counted.
- Each model failed in a different way (missed requirements, integration errors, or unverified assumptions), so a single generic code-review checklist will not catch what a specific coding agent actually tends to get wrong.
- The sample is small (ten disclosed tasks), the private codebases cannot be independently inspected, and the benchmark's publisher sells private company data and code to AI labs, all reasons to treat this as a data point, not a verdict.
- Pilot any coding agent on your own codebase, track cost per resolved task rather than sticker price, and match your review process to the tool you actually use, before committing budget.
Where ATLACIS can help
Sources
- Y Combinator: Launch YC, Real-SWE, a coding benchmark built from private company codebases (Janak Sunil, September 10, 2026)
- Specific Labs: Real-SWE benchmark and leaderboard
- samcodeman.com: Real-SWE benchmark tests private enterprise codebases (Samir Sengupta, September 12, 2026)
- THE D*AI*LY BRIEF: Fable Cost $18 a Fix on Private Code. Flash Cost $8. (Rajesh Beri, September 13, 2026)