This is an interesting and very detailed study. We need to be careful to measure objectively – are these tools helping us or not? AI coding can be addictive, where we get caught in a loop, prompting and waiting for it to do stuff – we think we are almost there, but it ends up taking many more cycles than we had anticipated. It often helps to go back to the basics and fully understand what we are doing.
We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation.