Skip to content

CI/CD for gateware at CERN

Introduction

This section deals specifically with CI/CD for gateware designs in the CERN context. Please read the general introduction to CI/CD first. It is often the case that pipelines for gateware projects need the running of proprietary tools, e.g. for synthesis and simulation. CERN purchases a limited number of licences for those tools, and is not at liberty to share them with people outside the organisation. This means pipelines using proprietary tools cannot run in public collaboration spaces like github.com and gitlab.com. The current recommendation is to run the CI/CD pipelines which require proprietary tools at gitlab.cern.ch, CERN's private GitLab instance. See the authoring and hosting section to learn how to mirror your project from a public collaboration space into gitlab.cern.ch using the "pull" method. Your CI/CD using proprietary tools can then run in your local mirror.

GitLab CI

The basic ingredient of CI is a 'pipeline'. A pipeline is a sequence of scripted steps that will be executed on the code in the repository. In GitLab, a pipeline, like all CI-related configuration, is defined in the file .gitlab-ci.yml which is placed in the root directory of the project repository. GitLab automatically detects the YAML file and initiates the process of executing the GitlabCI script depending on configuration, typically when changes are pushed in a certain branch, or on a merge request. GitLab CI is an accessory to the project code and in no way attempts to modify/append anything in the project repository. Any file created during the process of execution of a CI script which is worth keeping is called an artifact and is stored separately from the project repository.

Why you should use GitLab CI

Using CI has several advantages. Below is a short list, with some examples illustrating its usefulness in the context of gateware development.

  1. Centralisation
  2. Unsupervised build/tests. CI can be used for:
    • Directed tests (Cocotb)
    • Formal tests (Yosys)
    • HDL linting
    • Build and synthesis (GHDL)
    • Building documentation (Doxygen)
    • ...
  3. Lowers barrier to entry
  4. Saves results (artifacts) for later inspection
  5. Simplifies code review

How does it work?

As a result of a user interaction (push, merge request,...), a CI pipeline is triggered. This starts different tasks (named "jobs"), grouped in different stages. The stages could be, for example, "build", "code quality", "tests", and "package". The jobs are run on a "GitLab Runner", which may be any network-accessible computer which is correctly configured. They may be administered by CERN, or by yourself.

Three elements are necessary for using GitLab CI:

  1. Project-specific configuration (.gitlab-ci.yml): describes the tasks to run and when they should be run.
  2. Container Images: provide the binaries used for the tasks' execution (e.g. Vivado, Petalinux, GCC, ..).
  3. GitLab Runner: an application that runs the project's jobs using the images.

While (1) must be done on a per-project basis, (2) and (3) may use CERN-provided resources.

An important point of entry is GitLab's own documentation, available at docs.gitlab.com. Specifically, the CI/CD documentation is available here. CERN's own GitLab documentation for CI/CD is provided here.

CERN-provided Container Images

Many images for HDL development tools are available in this GitLab repository. The repository includes documentation on how to use them.

CERN-provided GitLab Runners

While CERN provides GitLab runners, one cannot expect images of HDL tools to work with them, due to the very large amount of disk space (2GB - 70+ GB) required to make them run. Because of this, using these runners is for now discouraged. Instead, HDL developers should use custom runners running on their own computers/VMs.

Custom Runners

In order to set up a custom runner, you need to have access to a computer/VM that matches you job's requirements. We recommend using CERN OpenStack with an XL image. The GitLab Runner must be installed and configured, following the instructions, and then registered for your particular project.

Custom images

If the CERN-provided Container Images are not sufficient for your needs, you may need to create your own container image. For this, docker is typically used. Its documentation is available here. Once the image is created, it must be made accessible to the runner. This can be done through the use of registries, or, for privately owned runners, by importing/building your image on the corresponding VM.