Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer

This is an interesting and very detailed study. We need to be careful to measure objectively – are these tools helping us or not? AI coding can be addictive, where we get caught in a loop, prompting and waiting for it to do stuff – we think we are almost there, but it ends up taking many more cycles than we had anticipated. It often helps to go back to the basics and fully understand what we are doing.

We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation.

Many crazy people say that developers are getting 10% more efficient every few months. That seems crazy high to me, but I will concede that the potential for AI to suddenly provide significant efficiency gains is out there – coming soon to a data center near you.

In the meantime, AI is just a nifty tool that sometimes helps and is sometimes harmful. I think there is a net benefit, but it’s not game-changing… yet.