Code Review Handbook

Previously published as a PDF

Code Review Handbook: Code Review Basics

How do you do a code review? Typically, I get an email from bitbucket or Github saying “Adam added you as a reviewer on ‘SMB-1182-larger-buttons’”.  This is probably the 10th PR of the day, it is a constant stream. Often there is a 20 file PR that I just do not want to get started on. This is the industry standard PR environment, where the requests are constant and each PR blocks the writer until he gets the magical number of review checkmarks that let him move that JIRA ticket to ‘Ready for QA’. 

The modern Pull Request workflow is what inspired me to write this handbook. This chapter will go over the basics of doing a code review. What the common expectations are, and what you should do to help your team move forward. 

The basic workflow for a Pull Request (PR) is that a member of the development team has completed a feature in a branch and now has to get ‘approvals’ from other team members before it can be merged into the release branch. These days it is common to require any code that gets used first go through an approval process. As a team member you are expected to review pull requests in a timely manner. Typically, you can put off PRs until the end of the day and hope that they get merged before you do anything. But its unlikely that you will escape reviewing entirely. Some teams require every team member to approve every pull request others require only one or two approvals before the branch can be merged. 

When you get down to reviewing code, the basic expectation is that you will read all of the proposed changes and comment about any bad practices or troublesome logic that could cause problems down the line. The dream is that you will catch a mistake or a security vulnerability that otherwise would have cost the company millions of dollars had it gone into production. The reality is that most of your comments will be reminding people to follow the style guide or to not organize their code like monkeys with type writers. In my opinion pull requests are rarely read carefully enough to catch bugs or vulnerabilities. 

The issue is that doing pull reviews throughly requires a substantial amount of time. A twenty file pull request, the like of which you might get two or three a day, could eat up thirty minutes by itself. Most developers have a lot of features to add and bugs to write, time spent reviewing other people’s work does not add a lot to your bottom line. 

Preparing to do Code Reviews

I get a lot of PRs to review every day and what can happen is you get in to work and there are 2 PRs waiting for you. You just sat down at your desk and there is already 20 pages of diffs for you to review. Then you get to the status update meeting, team members with outstanding PRs ask people to review them. Then your boss sends you an email with architecture questions, or more likely someone else relays your boss’s architecture questions to you via slack. You quickly hash something out to answer your bosses questions, then try to squeeze in a few minutes on your actual ‘work’ deliverables. An hour later its time to grab lunch before the rush starts, and head back to the office for the pre-backlog grooming meeting. During the meeting you get several urgent slack messages about the website being down and you spend the next couple hours fixing the issue. Now its 5pm and the 2 outstanding PRs you had at the beginning of the day have grown into 5 PRs. Your chats are full of teammates’ request for you to review their code already. 

Everyone has had a similar experience, where the list of PRs you need to review keeps growing and it does not seem like there is anyway to handle it. 

Block off time

The best way to handle Pull Request reviews is to block off time everyday to address them. I recommend a 30-60 minute block first thing in the morning. I usually have a 45 minute block before our daily standup which I use to review PRs and catch up on my email. Depending on how many PRs you need to review, consider scheduling another block of time at the end of the day. It could be 4:30-5pm or 6-6:30pm, depending on your schedule. The goal is to ensure that you are not stopping other people’s work by getting to PR reviews in a timely manner. 

Decide how much time is enough

If you have blocked off 30-60 minutes everyday and it still is not enough, I recommend cutting off PR reviews after you spend a certain amount of time on them. If you schedule one hour a day, stop reviewing PRs after that hour and do not review anymore until the next day. You will want to discuss this with your manager and team so that they know how much time it is taking to review all of the code. Your team may need to have a general discussion about how many reviews are needed for each pull request. You might have to change from the entire team reviewing every PR to only requiring two approvals for each pull request. Try to think about the Pull Request Review process as a tool that we use to avoid bugs and improve product quality. We do not ‘have’ to do Pull Request Reviews to have a quality product or avoid bugs, we want to use the tool to help us. So if your team is spending too much time reviewing Pull Requests it may be the right decision to cut back. 

Decide how Thorough Reviews should be

In addition or alternative to setting a maximum time spent on reviews, you can choose to do less through code reviews to save time. I would recommend having less people review each PR before you cut back on thoroughness of the reviews, because people tend to skip through reviews when they see that a few people have already approved the PR. Its better to have two people spend 15 minutes each then to have 5 people spend 3 minutes skimming the pull request. 

Setup your Code Review Environment

Once you have sorted out how much time you need, its time to setup your environment. To do code reviews you want to set up a split-diff view of the code. This can be done in several ways, Github, Bitbucket and most other central git repositories include a split-diff setting for Pull Request reviews, but it is not always the default setting. Find the documentation for your Pull Request tool and figure out how to change it to split-diffs by default. Its definitely worth trying for PRs with a lot of diffs in each file. 

There are a couple offline tools that can give you split-diffs if your central git store does not include that feature or if you are not using one of the common SAAS git services. Git includes a git-difftool by default which provides several different tools to help you view diffs, https://git-scm.com/docs/git-difftool. 

I also recommend getting a large screen to review pull requests in. A 13in laptop is going to make it harder for you to see changes and require more scrolling. Get a bigger screen if you can. 

Do the review in one go

I recommend doing code reviews in one go. It is a common temptation to squeeze PR review into short blocks of time, but then you will not be able to get through long PRs in one go. The last thing you want to do is skim through a long PR three times without understanding anything in it. Make sure you have at least enough time to read all of the changes and comment a bit before your next obligation. 

Three levels of Code Review

Not every code review needs the same level of thoroughness, sometimes just a quick check for obvious errors works. Other times you may need to throughly examine a major change and ask for multiple revisions. In this section we will describe three different PR review strategies you can use depending on the situation. 

The Quick Skim 

  • read the code
  • check for tests
  • ask questions
  • suggest improvements
  • get out

The Quick Skim is the fastest version of the PR review. You can use it when you are reviewing code that has already been reviewed several times or when the PR is small and unlikely to cause problems in your judgement. This is also a good option if you want to save time. 

The first step is to read the code, you want to read through each file of changes. Resist the urge to read as quickly as you can. You can read through most PRs in less than 5 minutes without rushing. If the PR is really too long to read in 5 minutes, its ok to take more time, but that should trigger a conversation with the team. Large Pull Requests are harder to understand and harder to integrate into the rest of the codebase. Sometimes you need to make sweeping changes, but you should not be seeing PRs like that everyday.

Next, check for updates to the testing suite. Most PRs should include new tests, or updates to the existing tests. If the PR is a bug fix, ask if they have a test to check if the bug is reintroduced. Its rare to see a PR that does not need any test changes at all. 

Ask questions about the changes. It could be formatting, reimplementing a method instead of using a library or just something that you don’t understand in the change. Finally, suggest at most a few of the improvements that you thought of and move on. You want to suggest only the most important changes that occurred to you. Do not give a list of twenty things you want changed in the PR comments. Pick the top three changes that will have the biggest impact on improving the code. The goal for the Quick Skim is to save you as much time as possible, and arguing or even talking about low impact changes to a PR is not going to save that time. Finally, approve the PR and stop thinking about it. Ideally you noticed some issues, pointed them out and they were addressed promptly. Average case, you pointed out some issues, one of them was addressed and you got back to work on your tasks with minimal interruption. 

The thorough review

  • read the code 
  • ask questions
  • suggest improvements
  • review tests carefully
  • consider implications
  • suggest missing test cases

The Thorough Review is targeted for subject matter experts reviewing changes to code that they know well. If you are the ORM guy reviewing a PR that makes changes to the ORM system, the team is probably relying on you to find any problems at the PR phase. Once you approve the PR other people are likely to just hit ‘approve’ and move on, so its important for you to catch the issues. 

The flow for the Thorough Review starts off just like the Quick Review, but our mindset is different. We are focusing on finding issues in the code instead of saving time. Start off by reading the code and asking questions about the rationale behind the changes. Suggest improvements or alternative approaches if you think of any. You do not necessarily have to force a change, but its useful to have people consider alternatives. 

Now, review the tests carefully. Do the test changes match the code changes? Are there any missing test cases that could have been added? 

The exhaustive/critical review

  • read the code 
  • download the code
  • run unit tests
  • boot up the application and manual test
  • ask questions, point out missing functionality 

The Exhaustive review, so named because you will be exhausted if you have to do a lot of these. The main difference between this one and the previous is that we are downloading the code and testing it locally. Checkout the branch and run the test suite, does it pass? Boot up a development server and do some manual testing, does feature x actually work with these code changes? Did the size of that button actually change? The idea behind the manual testing is to poke around and find anything that was not caught by the test suite. 

Three Levels Summary 

Quick 

  • read the code
  • check for tests
  • ask questions

Thorough

  • read the code
  • check for tests
  • ask questions
  • read tests thoroughly — check cases carefully 

Exhaustive

  • read the code
  • check for tests
  • ask questions
  • read tests thoroughly — check cases carefully
  • Run the code locally, test manually and via test suite

Links Post

Real time analytics article

https://adamsinger.substack.com/p/demystifying-real-time-analytics?utm_source=url&s=r

Danluu on the issues with purchasing solutions.

https://danluu.com/nothing-works/

Nuclear Stuff 

https://austinvernon.site/blog/nuclear.html

https://whatisnuclear.com/economics.html

Are we really engineers?

https://www.hillelwayne.com/post/are-we-really-engineers/

Configuration – Best Practices 

Static vs Dynamic Configuration

There are two main types of configuration static and dynamic. The main difference is in how the configuration is consumed by your services. 

Static configuration is immutable in the life time of an instance of your service. The service can read it once on startup and safely assume it will never change. 

Dynamic configuration is mutable in the life of a service instance. The service cannot assume the configuration it has in memory is still accurate. It must have a reliable way to get configuration changes. 

Implementations of static configuration 

Typically, static configuration is read on startup of your service and kept in memory for its lifetime. 

Config file in Service repository

The easiest way to do static configuration is to include a config file into your service repository. That way your service will always have access to the config file on startup. The config is put through the same code review process as the code. Plus code and configuration are kept in the same place making it easy to reference them against each other. 

The file can be anything from JSON, to a purpose built configuration language. 

Config file in config repository 

Here we simply take a config file we could include in the service repository and put it in a different repository. You could have one main config repository for a collection of services or a separate config repository for each service. 

Why would you do it this way? Splitting out the config from the code lets you deploy the config independently of the code. To release a config change you simply update the config repository then deploy it along your CI/CD pipeline. Finally, you trigger a fleet wide service restart to activate the change. 

In many setups this will be faster than deploying a config update in your service repository. 

Config file in external storage (S3, DynamoDB, distributed filesystem, SQL database, etc)

Keeping the config file in external storage has more tradeoffs than have the config in a repository. Now you have to consider whether the service has access to the external storage upon startup. If it doesn’t you may see repeated restarts none of which succeed. More stable external storage mechanisms are strongly preferred for static configuration. If the external storage goes down you may not be able to scale your system up or deploy new versions of the code. 

Implementation of dynamic configuration 

Dynamic configuration is typically implemented via polling and caching with timeouts. Your service might request config updates every five minutes from a centralized configuration server and update its internal state accordingly. Another option is to maintain a config cache and query the config service only when the particular key in question has exceeded its timeout period say 15 minutes. 

Technically, you could build a push based dynamic configuration system, but I haven’t seen it in the context of web services. As you add more and more services to your system having a client side cache helps keep load down and its easier to keep things in the cache up to date in a pull based system. 

Polling for updates 

A typical way to handle dynamic updates is to have your service regularly check with an external service to see if there were any updates. The results are usually stored in memory. Your service would make a request to its external configuration system on a set schedule from every five minutes to even every few hours. The schedule is set by your scaling needs and how quickly you need configuration to be updated across your fleet. If you want to roll out updates within ten minutes a polling time in the order of minutes is the way to go. 

Per request checking for updates

Another way to handle dynamic configuration is to check with the configuration server for every request. To make this scale you have to include a local cache of some kind. When your service receives a request it checks its cache and if the relevant configuration keys are expired it will make a request to the configuration service to get updates. 

How often will the configuration Change during good times?

The most important consideration for configuration is how often the configuration needs to change. Do you plan on changing it daily? Can it be updated weekly or monthly instead? What events would cause you to need to update it? 

Typically, if a human is generating the change your configuration can update to reflect the new state in hours or days. 

Dig into your use case. 

Are you getting configuration from a partner on a quarterly, monthly or weekly basis? In that case static config is the way to go. 

Are you getting rules updates from the product team on a daily basis?

 In that case static config is still the way to go. But this assumes you can deploy static config changes in a matter of hours reliably. If your team can’t deploy a change to static config reliably this situation might force you to use dynamic configuration. You might want to focus on improving your CI/CD before making the jump to dynamic configuration. 

Are you getting rules or other config changes on an hourly basis from the product team?

This is the inflection point for moving to dynamic configuration. Most teams do not have the ability to deploy changes to static config on an hourly basis. And typically even if you could pull it off dynamic config is a better solution here. 

How quickly does the configuration NEED to be changed in emergencies?

If you have an emergency will you need to update this configuration? Business rules probably do not need to be changed during a production outage. Instead you are likely to want to turn a feature toggle off. Or increase the timeout for a service call. 

How many services will use the configuration 

Having more services involved changes the tradeoffs you need to make when picking a solution. If you have just one service a static file or DynamoDB table can fulfill all your needs. Scaling that architecture might not work as well. You don’t want to move to a situation where you have dynamic configuration for one hundred services stored in one hundred AWS accounts across one hundred DynamoDB tables 

In organizations with more services consistency and automated rules enforcement become more important than convenience. Automatically confirming dynamic configuration conforms to standards across 100 unique service, AWS accounts and dynamodb table structures is likely impossible. Automatically confirming standards in one shared dynamic configuration service with one hundred client services is going to be a lot easier. 

Just tell me what to use!

If you want a general guideline my recommendation is to start with static configuration stored in your service repository. Once you have a serious need for rapidly updating configuration setup a DynamoDB table to store it in. But before you do so make sure that you actually expect to be making multiple configuration updates per day. If they can’t give you a list of the updates they want and the times they need them to go out, you don’t need dynamic configuration. 

Finally, if you have more than three services using dynamic configuration, setup an API service as your shared dynamic configuration store. 

SledgeConf July 15th 2022

4:30PM Pacific Time

https://sledgeconf.dev

Hey, I’m bringing back Sledgeconf on July 15th. We have two talks this time ‘Planning your Cloud Migration’ and ‘Practical Software Estimation’. 

“Planning your Cloud Migration”

Jay Manning joins us again to talk about planning and executing Cloud Migrations. Jay has extensive experience with AWS partners working through their Cloud journey. 

“Practical Software Estimation” 

I will be speaking on my research and interviews on the subject of practical software estimation. How do we estimate software today and what are the best practices?

Make sure to join the mailing list to get the invite link and blog updates! See you all in July!

https://sledgeconf.dev