Engineering leaders often assume that integrating AI coding tools will automatically accelerate development cycles. However, recent empirical evidence suggests that the reality is more nuanced.
A controlled study of experienced open-source developers indicates that AI assistance can sometimes hinder rather than help, leading to longer task completion times in real-world scenarios.
In short
- •
AI coding tools can increase task completion time by 19% for experienced developers compared to manual workflows.
- •
Benchmarks often overestimate AI capabilities by ignoring the friction of real-world context and human-in-the-loop requirements.
- •
Engineering teams should prioritize observability and human oversight to prevent AI-driven productivity regressions in complex codebases.
The Benchmark Gap
Standard AI coding benchmarks frequently rely on self-contained tasks that lack the complexity of large-scale repositories. These evaluations often prioritize algorithmic efficiency over the messy reality of existing codebases.
Because these benchmarks operate without human interaction, they fail to account for the small, iterative fixes that developers perform during standard workflows. This disconnect can lead to an overestimation of AI performance in production environments.
Real-World Productivity Trade-offs
The study recruited 16 experienced developers to work on open-source repositories with over 1 million lines of code. When these developers utilized AI tools, they took 19% longer to resolve issues than those working without AI assistance.
This regression suggests that the cognitive overhead of managing AI output—such as verifying suggestions or correcting hallucinated patterns—can outweigh the speed gains of automated code generation. For senior builders, this highlights the need for rigorous evaluation before scaling AI tools across engineering teams.
Source
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
