Skip to content

Continuous integration & delivery (CI/CD)

What is CI/CD?

CI/CD stands for continuous integration and delivery, or sometimes deployment.

Continuous integration (CI) is the practice of frequently building and testing a software system during its development. Continuous delivery (CD) is the practice of automatically publishing software updates. Continuous deployment (CD too, confusingly) is the practice of automating software deployments to production. While the concepts are distinct, they are often implemented using the same technical tools, and will be discussed together in the following.

Why CI/CD?

Setting up a continuous integration pipeline for a project is part of best practices, as it ensures that all code that is added to its repository satisfies basic quality requirements. The code is built automatically for each commit and the automated test suite is run, which allows to identify problems and errors immediately. This benefits all contributors to the project, but is particularly valuable to attract external contributors as it relieves them from setting up the whole toolchain to build and test the software locally which can be complex and time-consuming.

The typical steps in a CI workflow are:

  • linting and static analysis: checks on the source code to ensure that it is formatted according to the requirements of the project and that it doesn't contain detectable errors and vulnerabilities
  • building (for compiled languages): transforming the source code into an executable binary format
  • testing: running the automated tests to ensure that the new code doesn't break existing functionality and conforms to the specification embodied by new and/or modified tests

Continuous delivery further automates the software release process: e.g. updating the version, building the source code tarballs, publishing release notes, executables, etc.

Continuous deployment further automates the deployment of the released software to production systems.

Projects hosted at gitlab.cern.ch

CI/CD pipelines are defined in the .gitlab-ci.yml file at the root of the repository. More information, including CERN specificities, is available in the gitlab at CERN docs.

Projects hosted at github.com

On GitHub, CI/CD is implemented using GitHub Actions. Workflows are defined in the .github/workflows directory at the root of the repository. Extensive documentation is provided by GitHub.

GitHub-hosted runners

GitHub-hosted runners are available to execute the workflows. For public repositories, there are generous usage limits that will be sufficient for many projects.

For private repositories, a limited number of running minutes on GitHub-hosted runners is included in the free plan.

Self-hosted runners

Self-hosted runners allow workflows to be run on CERN infrastructure.

They can be used to circumvent the aforementioned usage limits, but they also offer the following advantages:

  • more accurate continuous benchmarking (less jitter)
  • easier access to CERN resources such as EOS or openstack object-stores
  • CERN authentication tokens not leaving CERN
  • CERN user authentication
  • significantly larger storage limits for artifacts / data

Danger

Self-hosted runners are not recommended for public repositories for security reasons. Make sure to carefully evaluate their need and implement the approach explained in the next paragraph.

Self-hosted runners present a security risk: external contributors can trigger workflows that run their own potentially malicious code using CERN resources and within the CERN network. If you really need to use self-hosted runners, then you must at least change the repository settings to require approval for all outside collaborators and carefully review all external Pull Requests before allowing the workflow to run.

One convenient way to run self-hosted GitHub runners at CERN is to set up a Kubernetes cluster and deploy the Actions Runner Controller on top of it.