You submit an estimate and are asked, "couldn't this be halved with AI?" You answer that you are using it, and the conversation moves on to "so, half then." The reason this exchange feels out of joint is that the difference between work AI helps with and work it does not is hard to see from the client side.
Material that shows that gap in numbers has appeared: a benchmark called Real-SWE, published in September 2026. Built by Specific Labs, a Y Combinator-backed company, it draws its problems not from public repositories but from private production codebases used with permission from real companies. Neither the code, the tickets nor the reference solutions exist anywhere on the public internet.
The results looked quite different from the public benchmark leaderboards.
38.8 percent at best
Eight frontier models were applied to ten tickets from real companies.
| Model | Completion rate |
|---|---|
| Claude Fable 5.1 | 38.8% |
| GPT-6 Astra | 33.8% |
| GPT-5.6 Sol | 16.2% |
Even the best falls short of 40 percent. On top of that, one of the ten tasks (a problem involving stream processing of analytics data) failed across all 64 attempts by all eight models.
The codebases used are ones actually running businesses, including a consumer product with more than 200,000 users and a financial platform that has processed more than 100,000 bank statements.
A few caveats up front. Ten tasks is statistically small, and the figure was published by the benchmark's own provider. Reading this number straight as "AI's real ability is 38.8 percent" would be crude. What to look at here is not the absolute value but the structure: where the gap from public benchmarks comes from.
The gap shows up in how many files one fix reaches
The number of files a Real-SWE solution touches has a median of eleven. Public benchmarks are said to sit at around six, making this roughly double.
Anyone who has worked on business systems will recognize the shape of this gap immediately. A real fix rarely ends in one place. Input validation, how the data is held, the handling on the aggregation side, the screen display, migrating existing data, and then testing the blast radius. One line of specification is scattered across layers.
On top of that, the information it needs sits outside the code. Premises like "this field has not been used since the 2019 rule change" or "this branch exists only for one particular client" are either written nowhere, or written in meeting minutes from three years ago. Specific Labs estimates that 99 percent of the tokens handled at real companies are invisible to frontier models. Whatever the precision of that figure, the direction — that there is overwhelmingly less material the model can read — is right.
Public benchmark problems differ on exactly this point. The repository is public, the history remains in issues, tests exist, and the scope of the change is relatively closed. For an AI, the material is all there.
What the client side can reflect in estimates
The practical conclusion to draw from this is not "AI is unusable." It is that you change how you read an estimate depending on whether the conditions under which AI works exist at your company.
Let us separate the work AI helps with from the work it does not.
- Works well: parts built from scratch, adding tests, routine conversion and migration scripts, explaining existing code, first-pass reading for reviews
- Works poorly: specification changes to long-running business systems, areas with many implicit premises, changes that require agreement across multiple teams, and defects with unknown reproduction conditions
When these two are mixed together in an estimate, the "half with AI" discussion never connects. Show separately which steps AI helps with and which steps still take the same human time, and the conversation becomes concrete. The danger of discussing development productivity as a single number is covered in the pitfalls of measuring generative AI's impact with metrics.
There is also a cost on the side using the AI: the usage charges for repeated attempts, the time of the person checking the results, and the effort of reverting failed changes. This overlaps with the ground covered in counting the invisible costs before building an AI agent platform and how to keep AI coding running costs down. Estimating on the premise that if the completion rate is 40 percent, people pick up the other 60 percent is the realistic approach.
Getting your own code into a state AI can read
Even with the same model, results change when there is more material it can read. There is preparation the client side can do in-house.
- Write the premises of the specification in one place. "Fields no longer in use," "branches for specific customers," "the history behind past rule changes." You do not need a perfect design document. Even leaving bullet points before the person who knows the history is gone makes a difference
- Add more tests. A state where you can mechanically confirm that an AI's change has not broken existing behavior is worth the same to people as it is to AI
- Make the boundaries explicit. What is in scope and where you stop touching things. A request that is vague about this causes rework even when you give it to a person
All three are less an investment for AI than an investment in being maintainable. Thinking of it in that order — AI happens to work better as a result — also makes the cost easier to explain.
A benchmark number is not your number
Finally, a word on how to handle this kind of score.
The completion rate is not determined by model performance alone. It varies greatly with what tooling it was run under: the range of files it can reference, whether it can run tests, and how many retries it gets after a failure. The same model produces different results in a different environment. Declaring internally that "AI can only do 40 percent" and expecting that "the latest model will be fine" are both wrong for the same reason.
If you want to measure it at your own company, the method is simple: pick three issues you closed recently and have it reproduce them using only the information available at the time. Because the correct answer is known, evaluation is easy. How many of the three pass, and what was missing in the ones that did not? That becomes your number.
While you are at it, note down why each one failed: "there were no tests so it could not verify," "it could not find where the related processing lives." That list becomes, as is, the list of places to shore up next.
What to do next
Think of your three most recent change requests and ask your developers how many files each one touched. If around eleven files is normal, it is reasonable to set modest expectations for how much AI will shorten the work.
Then list the "specifications only one person knows." That is the part you can hand neither to an AI nor to a new team member. Start with the work of putting it into a form that can be handed over, and how well AI works will change too.
At GleamHub, we handle consultations on organizing the specifications of existing systems, separating out where AI-assisted development applies, and migrating step by step toward a maintainable state, through our development, AI and automation advisory services. Because the approach depends on your system's structure and history, please get in touch via Contact.
