The capability cost behind AI coding assistants

A randomised trial published on arXiv in late January 2026 sat under the title How AI Impacts Skill Formation (Shen & Tamkin, 2026). It compared two groups of developers learning a new asynchronous programming library. One group used AI assistance. The other coded by hand.

The headline result, from the abstract: AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Anthropic's public summary of the same work reports the AI group averaged 50% on a comprehension quiz versus 67% for the hand-coding group, with the biggest gap on debugging questions. The full paper is a working paper, not yet peer reviewed, and the authors are affiliated with an AI lab. Treat the numbers as directional. Treat the direction as serious.

If you run delivery for an engineering organisation that has rolled out AI coding assistants over the last 18 months, that result is a risk worth tracking now, not after the next reorg.

Two things at once

The current AI conversation tends to pick a side. One claim says coding assistants deliver large productivity gains. The other says they weaken the next generation of engineers. The evidence in front of us supports a narrower and more useful reading.

Throughput gains are real for routine tasks where the developer already has mastery. They tend to evaporate or invert on genuinely unfamiliar work. Shen and Tamkin found no significant average efficiency gain in the new-library trial. Vella and Blincoe's longitudinal study of AI coding assistants (2026), which I used in a previous piece, reports the same pattern from the other side: developers shift time toward review, verification, and correction once AI generates most of the first draft. That study is a preprint and rests on developer self-report rather than objective performance data.

At the same time, comprehension and debugging deteriorate when the cognitive load moves into the tool. Shen and Tamkin show this for code. Lee and colleagues from Microsoft Research, in a CHI 2025 paper on 319 knowledge workers and 936 first-hand examples, show the same mechanism in general knowledge work. The measures are self-report. The moderator they isolate is unsurprising once you read it: the more a user trusts the model, the less critical thinking they do. The more they trust their own grasp of the task, the more they do. Confidence in the tool reduces engagement with the problem.

These two findings, read together, do not condemn AI assistance. They describe its shape. The shape is what a Head of Delivery has to design around.

What the license changes

A coding assistant license is a productivity instrument. A coding assistant deployed without a workflow design is, after about 18 months, a capability instrument too. It quietly chooses which kinds of mastery your team accumulates and which it skips.

Verwijs and Russo's Theory of Scrum Team Effectiveness (2023, ACM TOSEM) names the team-level drivers that actually move delivery outcomes: responsiveness, stakeholder concern, continuous improvement, management support, team autonomy. None of those drivers improves when each developer produces more lines per day. Several of them depend on capabilities that grow through the painful work AI is best at removing: reading unfamiliar code, debugging a system you do not yet understand, holding a model of someone else's domain in your head long enough to argue with it.

That is the capability cost. It does not show up in any sprint metric. It shows up later as an organisation that ships fast on routine work and stalls on anything new, because the people who should be operating without the model never finished learning how.

The mechanism

The reason this matters is that the cost is a system property, not a developer property. Shen and Tamkin's second finding is the operationally interesting one. They identified six interaction patterns. Three of them preserved learning even with the AI present: asking follow-up questions, requesting explanations of the generated code, posing conceptual questions while coding independently. The other three did not.

Capability does not depend only on whether the team uses AI. It depends on how. And how is something a delivery system can shape: review policy, onboarding design, work assignment, and what the team considers normal.

This is also where the delivery lever is. You do not need a corporate policy debate about AI use. You need a workflow design that distinguishes two kinds of work: tasks where the human already has mastery and the goal is throughput, and tasks where the goal is mastery and AI is allowed only in cognitive-engagement modes.

What changes on Monday morning

Three concrete moves, all small enough to do without a new budget line.

1. Tag work by familiarity, not just size. Routine, well-understood work types (bug fixes inside a known module, boilerplate, documented refactors) are the low-risk delegations. Unfamiliar work (a new library, a new domain, code the team did not write) is where delegation costs capability. Mark them in the tracker. Two tags is enough.

2. Define an explicit AI mode for unfamiliar work. No corporate ban. A workflow rule: on unfamiliar work, the engineer writes the first attempt before opening the assistant, then uses it for explanations and follow-up questions, not for a generated draft. Shen and Tamkin identified this shape; it is testable and observable in pull-request behaviour.

3. Measure one capability proxy alongside throughput. Pick one. Time-to-confident-merge for a new hire on an unfamiliar service. Median debugging time on incidents owned by engineers with under two years on the codebase. Review-comment depth on PRs that touch unfamiliar areas. None of these is perfect. Each of them is harder to game than story points and easier to act on than developer experience surveys.

The reason these moves work is the same reason the original problem exists. Capability formation is a system output. It responds to the work the system actually routes through human cognition. If you route every unfamiliar piece around that cognition, you may save time now and accumulate a capability debt that no metric on your current dashboard is built to see.

What to defend

If the only argument for adopting AI coding assistants is short-term cycle time, the defence is weak. Shen and Tamkin did not find average gains on unfamiliar work. Lee et al. found the cognitive cost is real and predictable. The work shifts; it does not vanish.

The more defensible position in budget discussions is this: AI assistance is a productivity instrument for routine work where the team already has mastery, and a capability risk for unfamiliar work unless the workflow forces cognitive engagement. The cost of getting that distinction wrong is paid later, not this quarter.

A Head of Delivery who can say that sentence clearly has a stronger case than one who only argues with cycle time.

Sources

• Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (pp. 1–22). ACM. doi.org/10.1145/3706598.3713778
• Shen, J. H., & Tamkin, A. (2026). How AI impacts skill formation [Preprint]. arXiv. arxiv.org/abs/2601.20245
• Vella, A., & Blincoe, K. (2026). The impact of AI coding assistants on software engineering: A longitudinal study [Preprint]. arXiv. arxiv.org/abs/2605.23135
• Verwijs, C., & Russo, D. (2023). A theory of Scrum team effectiveness. ACM Transactions on Software Engineering and Methodology, 32(3), Article 74. doi.org/10.1145/3571849