This time I will show you some techniques I used to reduce a Django + React app Continuous Integration (CI) pipeline from 15 minutes to just 3 minutes.
It’s quite common to end up with a CI pipeline that runs for 30+ or even 60+ minutes. At some point you’ll need to invest some time to speed it up.
The exact steps to achieve these speed ups are different for each CI provider.
One of the biggest time savers is reusing package data fetched from previous builds when the packages haven’t changed.
- Cache your
yarnpackages. Checksum of
- Cache your
pippackages. Checksum of
Pipfile.lock(Pipenv) as key
Caching is a balance between reliability (not using an out-of-date or inappropriate cache) and getting maximum performance (using a full cache for every build).
Alpine images are lightweight and an obvious choice for a base image, but sometimes it’s better to have a beefier base image that has most dependencies and language tools already pre-installed.
For example installing
numpy to an Alpine
based image requires compiling them from source files, which
can easily take over 10 minutes StackOverflow discussion.
Figure out the tasks that can run independent of one another.
If you have access to multiple runners, use parallelism. Large test suites can even be split across multiple machines. For example CircleCI supports automatic test allocation across containers.
Take a look at the times of each job. Can the longest running job be split into multiple smaller ones?
Let’s say there’s a task
yarn lint, which runs for 15 seconds and
yarn test, which runs for 40 seconds.
yarn install as a prerequisite (which takes 60 seconds). The trick here is that the installed
node_modules folder can be shared between the workspaces.
So instead of running
yarn && yarn lint && yarn test in one job,
consider two jobs with commands
yarn && yarn lint and
yarn && yarn test.
Speeding up CI is an exciting challenge. Analysing and figuring out the slow parts is the key. You may even end up going down a rabbit hole just to shave off a few more seconds.