🚀 Speeding up our deploys by ~35%

Having a Continuous Integration/Continuous Deployment pipeline is the industry standard nowadays. CI/CD is great and has a very well documented list of benefits. However, one of the drawbacks is how slow the pipeline can become as your codebase grows.

At carwow, we have already taken many steps to try and ship faster, without compromising on the reliability and safeguards that a good CI/CD pipeline offers us.

Some of the things we do involve:

• Precompiling our assets in order to stop ad-hoc compilation occurring while our test suite runs
• Extensive use of caching on assets and dependencies
• Parallelisation of our test suite
• Optimising dependencies between jobs in our workflow (i.e. workflow steps should start as soon as they can - they should not wait for any step that they do not rely on)

All of these optimisations have helped to increase the speed of our CI/CD pipeline, and in turn enabled us to get changes - and fixes - out the door that much quicker. These optimisations are worth it because we deploy fairly regularly; anything between 15–35 times a day.

That being said, there is always some room for improvement. All of our production applications follow the same general pattern for their workflow (and I assume this is fairly universal):

1. Install dependencies
2. Compile assets and run code linters
3. Run our test suite
4. Deploy

A quick look at recent builds on our main applications showed that we spend a lot of time during our deploy phase. But why?

A brief overview of Heroku’s slug compilation and release process

We deploy all our applications to Heroku and make use of their buildpacks in order to compile our source repositories into slugs that Heroku can run.

The de facto way of deploying to Heroku is by pushing to their git remote. Doing so will lead to the following:

1. Receive new commit
2. Sequentially run each buildpack via git hooks. In general, this involves installing dependencies and compiling code/assets
3. Bundle up the result of building into a slug
4. Run an optional release command
5. Restart all dynos with new slug

How we sped up our deploys

Now that we understood a bit more about what exactly Heroku was doing, we could start to find out why it was taking so long and come up with a solution.

Earlier in this post, I mentioned that one of the ways in which we have previously improved our build times was by “Optimising dependencies between jobs in our workflow”, which means ensuring we aren’t making any steps wait for things they don’t depend on. Our previous deployment process was a great example of us not doing this.

Deploying to Heroku is split into two steps bundled together: build and release. The release step has a dependency on our test suite passing - we don’t want to release any code that doesn’t pass our tests. The build step, on the other hand, does not. The compilation and release of the slug have different pre-requisites to execution.

Conceptually what we want to do is start the time-consuming process of compiling our slug as early as possible and separate it from the release step. We just had to find a way to do it.

There are a few ways of decoupling the build and release when using Heroku. There are two main alternatives that they recommend: Deploying with Docker, or building your own tarball.

These two methods are quite involved and would likely mean reimplementing Heroku’s slug compiler, as well as changing the way in which we configure which buildpacks we use, and most likely a bunch of other stuff we hadn’t even considered.

There had to be a simpler way. There must be a way in which we can run Heroku’s own build process without releasing the slug to production, right? This would allow us to kick-off building a new slug and wait for our test suite to pass, at which point we can simply promote the slug to production.

Unfortunately, there is no way to do that, at least directly. If you dig deep enough into Heroku’s documentation you will find this:

So that’s what we did! We now do a normal Heroku deploy via git, except to a non-public instance of the app. This runs through the normal build process and performs an empty release. When - and if - the test suite succeeds, our deploy step uses [Heroku’s Platform API] to find the appropriate slug from our compile app, and promote it to the production app. Our script looks something like this.

All of this means that our pipeline now looks like this (notice the compile_slug steps in the bottom left).

This allows us to save a significant amount of time in our builds by parallelising the slug compilation process to run while our test suite runs - we’ve seen speedups of 30–45% on our master branch builds. And all it took was some configuration changes and a little bit of bash 🎉.

We use Terraform to manage our infrastructure, which made the process of creating a new (free, zero dyno) Heroku application for each production app simple; just a change in a variable once we had initially configured it.

There are some things to watch out for though:

• Make sure that you don’t run any release commands you shouldn’t on the compilation app. We did this by short-circuiting the release command with [ ! -z $COMPILE_APP ] || release
• You may need to attach databases, other add-ons, and set some environment variables on your compilation app in order for it to work; in Rails at least, asset compilation requires loading your whole application.

I hope this information is useful and that you might also attempt the same or a similar approach in order to speed up your deploys, and hopefully, you see the same speedups as we have! If you know of any other interesting alternatives or decide to start your own project that enables building tarballs/slugs based on a list of buildpacks, please let me know in the comments.

This post was originally published on “carwow Product, Design & Engineering” on Medium (source)