👋 Welcome!
This time I will show you some techniques I used to reduce a Django + React app Continuous Integration (CI) pipeline from 15 minutes to just 3 minutes.
It’s quite common to end up with a CI pipeline that runs for 30+ or even 60+ minutes. At some point you’ll need to invest some time to speed it up.
The exact steps to achieve these speed ups are different for each CI provider.
Cache Dependencies
One of the biggest time savers is reusing package data fetched from previous builds when the packages haven’t changed.
- Cache your
yarnpackages. Checksum ofyarn.lockas key - Cache your
pippackages. Checksum ofPipfile.lock(Pipenv) as key
Caching is a balance between reliability (not using an out-of-date or inappropriate cache) and getting maximum performance (using a full cache for every build).
Choose the right Docker Images
Alpine images are lightweight and an obvious choice for a base image, but sometimes it’s better to have a beefier base image that has most dependencies and language tools already pre-installed.
For example installing pandas or numpy to an Alpine
based image requires compiling them from source files, which
can easily take over 10 minutes StackOverflow discussion.
Concurrent jobs
Figure out the tasks that can run independent of one another.
If you have access to multiple runners, use parallelism. Large test suites can even be split across multiple machines. For example CircleCI supports automatic test allocation across containers.
Optimize by the slowest job
Take a look at the times of each job. Can the longest running job be split into multiple smaller ones?
Let’s say there’s a task yarn lint, which runs for 15 seconds and yarn test, which runs for 40 seconds.
Both have yarn install as a prerequisite (which takes 60 seconds). The trick here is that the installed node_modules folder can be shared between the workspaces.
So instead of running yarn && yarn lint && yarn test in one job,
consider two jobs with commands yarn && yarn lint and yarn && yarn test.
Conclusion
Speeding up CI is an exciting challenge. Analysing and figuring out the slow parts is the key. You may even end up going down a rabbit hole just to shave off a few more seconds.