Ian Mulvany

December 25, 2022

async code reviews kill throughput.


This video about continuous code review is excellent https://m.youtube.com/watch?v=fYFruezJEDs

The main takeaway is that in order to get reasonable throughout along with retaining code quality, you need to look to techniques of cop creation (pairing or mobbing). 

Taking a economic systems approach to what happens with pull requests the speaker shows: 

•  Written feedback (eg comments on pull requests) is high latency low throughout feedback. It costs a lot to write and does not provide as much information as a conversation. 
•  The amount of feedback drops, the larger the pull request. 
•  Wait time increases as a reverse function of pull request size. 
•  Inventory expands until it matches the wait time in the system. 

There is an interesting analysis of the economics of pull reviews. While shorter pull requests are easier to review, if you have more of them they create more pull requests, creating an increase in interruptions. 

Basically, with asynchronous feedback there is a trade off between throughout and quality. 

In order to not exponentially loose the throughout while reducing the average size of the pull request, people need to get exponentially closer and closer in time. 

The findings in this talk stand in contrast to those in the post I just wrote about the work of James Evans and his team. In that work for scientific output, you want to reduce coordination between teams in order to get better results. In this analysis it’s about increasing coordination between developers. I guess the former model would be one where you had really hard programming problems, and you had all of your developers working independently on the same tickets, but that scenario doesn’t happen in a software development system. 

There are a couple of tests one could take to look at whether the following approach might work for your own org. 

•  can developers pair in a productive way? You need to have both good psychological safety, a good good culture of mentorship, and no knowledge silos in team members. 
•  How does throughput map to pull request size, and size and quality of pull request comments?