Modernizing your build pipelines

  • A high-quality pipeline must be fast. This needs quick feedback. To achieve this, let your CI tool parallelize all tasks that don’t have mutual dependencies and avoid running multiple checks together.
  • Have pipelines reflect in the code and call shell scripts that also work locally for easier testing before pushing to deploy, enabling a faster feedback loop.
  • To ensure your pipeline is reliable and reproducible, use containers to run each task in isolation and build the containers within the pipeline, a fresh container at each step.
  • While a persistent workspace saves time, it can build in flakiness, for which a good tradeoff may be improving speed by caching dependencies instead of downloading them each time.
  • Keep your pipeline highly visual and avoid over-abstraction. Visualization makes builds easy to understand and allows failed builds to be traced back quickly.
  • Your system must be scalable across multiple pipelines. Avoid duplication (slows the pipelines down) and parametrize tasks instead, so that you configure them by passing variables, and build a library of tasks that lets you reuse code across pipelines, while also reducing coupling between tasks and pipelines.

Full post here, 10 mins read