Presentation of the pipeline system

The Continuous Integration/Continuous Development pipelines are here to ensure the stability of the codebase. These are a set of operations that, unless specified below, are executed everytime you push on a repository. These operations are usually split in multiple stages, each stage containing one or more jobs.

💡 If you want to knnow when given stages are executed head to gitlab_shared_files/.gitlab/ci/rules.gitlab-ci.yml

Stages

Common stages

🛠️ build Build the project in multiple configuration to ensure its stability
- Some build configurations can be triggered manually and will be triggered in a non draft merge request otherwise.
🥊 test : Test the project

Linting and coverage stages

Automatically triggered in non draft merge request, can be manually triggered otherwise.

🔎 coverage : Check the parts of the code that are covered by the tests of the project.
📊 static_analysis : Lint the code to find bad practices or issues leading to undefined behaviors

📦 Packaging stages

These stages are specific to release and will be triggered when pushing on main.

🛠️ release : Package the project create a release : this includes building and testing the package on multiple platforms
🚀 deploy : Push the package on the package hub (usually pip)

Job example

For the build stage an example of job is building the project with pip install {project} as for another it will be building the project with cmake for a given compiler.

Definition of these operations

All the operations cited above are stored in .gitlab-ci.yml format which is just a fancy way to stored bash scripts. The basis of these operations is stored in a .gitlab-ci.yml file at the root of the repository.

As the whole pipeline requires a lot of code, this file usually refer to other files to ease the reading. They can either be local files or be located in another repository.

For the main aidge project the files are hosted locally in the project. For all other aidge projects (core, backend_cpu, ...), the files are located within the gitlab_shared_files repository.

Modify the pipeline

To modify the pipeline, follow the development workflow and read the README.md of the repo to know how to test your pipelines.

Understand the artifact retrieval process

The modules of aidge are not bound by any type of submodule linking. However, they need each other to work. Locally this is simple : build moduleA and then moduleB.

On the CI we need to find a way to connect each project. This is done by rertrieving the build artifacts :

Artifact definition : Artifacts are files that are saved for a given time after the execution of a job.

Interface explained

To call the function, we use the [!reference tags](https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags) mecanism. This allows to call on specific part of a given job set with the variable DEPENDENCY_JOB

- DEPENDENCY_JOB="$CI_JOB_NAME"
- !reference[ <os>:download:<object_to_download> , <use_list_dependencies_job> ]

Arguments :

os the job is running on : ubuntu / windows
object_to_download : job artifacts or the whole repository
use_list_dependencies_job : to choose the dependencies that will be retrieved. The <platform>:download:list_dependencies job contains default build dependencies for each repo. Usually it must be used except for case when a repository needs extra dependencies for testing (i.e. aidge_quantization). Values :
- before_script to use list_dependencies
- script to avoid it.

⚠️ If you use only the script, you will need to provide yourself a dependency list in the form of a $DEPS_NAMES variable that contains an array of the dependencies names

Short example : build on ubuntu.

I just pushed on project aidge_backend_cuda, the job currently running is build:windows_cpp, its job is to build the repository on the windows os.

aidge_backend_cuda needs dependencies to be built, I can choose to retrieve the whole repositories and build them but they have already been built in their respective pipelines. Hence, I can just retrieve their artifacts that contain the build/ with the built .solibraries.

We are in the build stage and default build dependencies are listed in <os>:download:list_dependencies, hence I need to use the before_script job.

My call will be :

- DEPENDENCY_JOB="build:windows_cpp"
- !reference [.ubuntu:download:artifacts, before_script]

Short example : missing test dependency on ubuntu

See aidge_quantization example here

⁉️ Why was this dependency not just added in list_dependencies ?

Because aidge_quantization compiles with cpp files, hence the artifacts to retrieve are from the jobs that build C++. But aidge_onnx is a pure python package (see below for more informations) that only builds with pip install, these jobs are named build:ubuntu_python.

Thus when looking for jobs named build:<os>_cpp.... the select_jobs job would have found nothing

Detailed example : inner mecanism

The pipeline is currently running the build:ubuntu_cpp job want for aidge_backend_cpu, this project needs aidge_core to build.

aidge_core has already been built in its respective pipeline, hence I only need to retrieve the artifacts that contains the build/ folder of aidge_core.

My call will be

- DEPENDENCY_JOB="build:ubuntu_cpp"
- !reference [.ubuntu:download:artifacts, before_script]

However there are many branches in the projects and we need to decide from which branch and which job we need to retrieve the artifacts from.

Choosing the dependencies to retrieve

By setting use_list_dependencies_job to before_script we chose to use the list_dependencies job, located here.

This will set dependencies to retrieve to ("aidge_core") .

💡 If you need to add a new repo : Its default build dependencies must be listed in this job.

Choosing the branch

There are 2 main cases :

You are working on a branch with no associated MR or with an MR at is in draft.
You are working in a NON-DRAFT MR.

Case 1

We look for branch a branch with a similar name in target repo (i.e. aidge_core). If we don't find one, we choose the devbranch.

Case 2

If a merge request pipeline IS NOT in DRAFT each dependency is retrieved from the merge request TARGET branch.

⚠️ In this case, The pipeline might fail if the following criterias are filled :

you just passed your Merge Request from DRAFT to NON-DRAFT

Your project has dependencies with unmerged branches with identical name as your current branch.

Choosing the job

We retrieve the latest SUCESSFUL $DEPENDENCY_JOB (usually the build job) on the branch chosen just before.

⚠️ One drawback of this method is that you need to wait for the other dependencies jobs to finish to launch your pipeline.

if you pushed simulataneously on aidge_core and aidge_backend_cpu the process won't work. Because the latest successful job will be the second to last job as the current job is running.

Release process

Its always done in 2 parts :

Creation of the package
Test of the package created. However the process depends on wether the package is a pure python package or not.

🐍 Pure Python package

A pure python package has no other language involved, not counting its dependencies

This package is created with the pip wheel command. Since python is a scripting language, this package is os/arch independent. As long as the platform has python installed, there is no problem.

🔠 Unpure package

Some aidge packages include C++ code, these compilations are platform dependents and need to be built n multiple platforms. This is why cibuildwheel exists. This projects generates several docker on chosen platforms to compile the code and create the package. Most of its configuration is done throught the pyproject.toml file in each project.

Most of these configurations are common. They include a specificity : the cibuildwheel_build_deps_before_build_wheel.* scripts Whose job is to build the aidge dependencies of the project in the freshly created docker before building the project.

Understand the CI CD Pipelines