ci: add documentation for the continuous integration system

Brand-new documentation for the four repositories: manifest, meta-ohos,
xts_acts and docs is provided in a format that fits the new documentation
project.

I chose to write it from scratch and avoid the extensive graphics
present in the "ci-docs" repository as I wanted to focus on the essence
first. I plan to follow up with documentation of the REUSE and DCO
check jobs. Separately I would like to explore limited diagrams where
they would convey additional benefits.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>

ci/docs.rst

+.. SPDX-FileCopyrightText: Huawei Inc.
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+The docs Repository
+===================
+
+.. _docs repository: https://git.ostc-eu.org/OSTC/OHOS/docs.git
+
+The `docs repository`_ contains the general documentation of the OpenHarmony
+project. The documentation is normally written in reStructuredText, with an
+important difference. Unlike in a regular project, the documentation here is not
+standalone. Instead, the git repository contains symbolic links that are valid
+when an entire workspace is constructed with ``git-repo`` and the `manifest
+<https://git.ostc-eu.org/OSTC/OHOS/manifest>`_  repository.
+
+This complicates the testing process and the deployment process, but allows the
+resulting documentation to span multiple repositories, permitting code and text
+to be conveniently changed in one go.
+
+Special Jobs
+------------
+
+build
+.....
+
+The ``build`` job ensures that the documentation can be built with `sphinx-build`
+without any warnings or errors. Apart from the complexity of setting up the workspace
+as described above, it is fairly typical.
+
+deploy
+......
+
+The ``deploy`` job builds aggregated documentation view that is rendered at
+https://openharmony.readthedocs.org/. This job effectively constructs the
+workspace as described above and then archives the entire documentation source code,
+de-referencing any links that were followed to other repositories, and commits the
+updated set of files to helper repository which is observed by readthedocs.
+
+The helper repository is called `openharmony-readthedocs-aggregated
+<https://git.ostc-eu.org/OSTC/infrastructure/openharmony-readthedocs-aggregated>`_.
+That repository contains a webhook, configured at the level of the GitLab project, which
+asks readthedocs to re-build the documentation.
+
+Implementation Highlights
+-------------------------
+
+Commit Credentials
+..................
+
+Commit to the aforementioned aggregated repository is allowed by a personal access token
+that is set up in that repository. The value of the token is stored as a protected variable
+available to the docs repository.

ci/index.rst

+.. SPDX-FileCopyrightText: Huawei Inc.
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Continuous Integration
+======================
+
+OpenHarmony ``git`` repositories hosted on https://git.ostc-eu.org/OSTC use
+GitLab pipelines for building and testing changes before they are merged.
+Individual pipelines are documented in each repository, but some general or more
+important elements are described below.
+
+.. toctree::
+   :maxdepth: 1
+
+   manifest
+   meta-ohos
+   xts-acts
+   docs

ci/manifest.rst

+.. SPDX-FileCopyrightText: Huawei Inc.
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+The manifest Repository
+=======================
+
+.. _manifest repository: https://git.ostc-eu.org/OSTC/OHOS/manifest.git
+
+The `manifest repository`_ contains the "map" to all the other repositories used
+by OpenHarmony and is thus important from CI point of view.
+
+The *manifest* repository also contains job definitions that pipelines defined
+in other repositories can include. Those are all stored in the `.ostc-ci`
+directory. Internally, in the manifest repository, the same rules are used to
+create a git-repo workspace, synchronize all the git repositories, initialize
+and configure bitbake and build bitbake recipes corresponding to supported
+reference boards.
+
+Shared Job Definitions
+----------------------
+
+The following GitLab job definitions are shared with other repositories. They
+can be customized at the level of a particular git repository, but in general
+constitute the set of supported build configurations.
+
+linux-qemu-x86
+..............
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=qemux86``. This job checks that OpenHarmony software can be built for
+a basic 32bit x86 virtual machine.
+
+The cache for this job is publicly available.
+
+linux-qemu-x86_64
+.................
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=qemux86-64``. This job checks that OpenHarmony software can be built
+for a basic 64bit x86 virtual machine.
+
+The cache for this job is publicly available.
+
+linux-qemu-arm
+..............
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=qemuarm``. This job checks that OpenHarmony software can be built for
+a basic 32bit ARMv7 virtual machine.
+
+The cache for this job is publicly available.
+
+linux-qemu-arm64
+................
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=qemuarm64``. This job checks that OpenHarmony software can be built
+for a basic 64bit ARMv8 virtual machine.
+
+The cache for this job is publicly available.
+
+linux-seco-intel-b68
+....................
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=seco-intel-b68``. This job checks that OpenHarmony software can be
+built for the SECO B68 development board, which contains an Intel x86_64 SoC.
+
+The cache for this job is not public, pending legal review of any firmware that
+may be included.
+
+linux-seco-imx8mm-c61
+.....................
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=seco-imx8mm-c61``. This job checks that OpenHarmony software can be
+built for the SECO C61 development board, which contains an iMX8 SoC.
+
+The cache for this job is not public, as it contains proprietary elements that
+cannot be redistributed without an agreement with Freescale.
+
+linux-stm32mp1-av96
+...................
+
+This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and
+``MACHINE=stm32mp1-av96``. This job checks that OpenHarmony software can be
+built for the 96Boards Avenger development board, which contains an STM32MP157
+SoC.
+
+The cache for this job is not public, pending legal review of any firmware that
+may be included.
+
+zephyr-qemu-x86
+...............
+
+This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and
+``MACHINE=qemu-x86``. This job checks that Zephyr can be built for a basic 32bit
+x86 virtual machine.
+
+The cache for this job is publicly available.
+
+zephyr-qemu-cortex-m3
+.....................
+
+This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and
+``MACHINE=qemu-cortex-m3``. This job checks that Zephyr can be built for a basic
+32bit ARM micro-controller virtual machine.
+
+The cache for this job is publicly available.
+
+zephyr-96b-nitrogen
+...................
+
+This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and
+``MACHINE=96b-nitrogen``. This job checks that Zephyr can be built for the
+96Boards Nitrogen development board, which contains an nRF52832 SoC.
+
+The cache for this job is not public, pending legal review of any firmware that
+may be included.
+
+zephyr-96b-avenger
+..................
+
+This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and
+``MACHINE=96b-avenger96``. This job checks that Zephyr can be built for the
+96Boards Avenger development board cortex-M3 core, embedded into STM32MP157 SoC.
+
+The cache for this job is not public, pending legal review of any firmware that
+may be included.
+
+freertos-armv5
+..............
+
+This job builds ``zephyr-philosophers` using ``FLAVOUR=freertos`` and
+``MACHINE=qemuarmv5``. This job checks that FreeRTOS can be built for a basic
+32bit ARMv5 micro-controller virtual machine.
+
+The cache for this job is publicly available.
+
+Special Jobs
+------------
+
+linux-glibc-qemu-x86_64
+.......................
+
+This job extends ``linux-qemu-x86_64`` and differs in the following way.
+
+This job performs a build with the libc switched to ``glibc``. It only runs on a
+schedule that is defined in the GitLab project settings for the `manifest`
+repository. In practice it runs daily to check if the Linux favour could be
+switched back to ``glibc``, from the default ``musl`` that is used right now.
+
+Implementation Highlights
+-------------------------
+
+Implementation details are stored directly in the manifest repository, as code
+comments. Only specific highlights are listed below.
+
+Local git-repo Mirror
+.....................
+
+The ``.workspace`` job relies on a git-repo mirror that is mounted into the
+execution environment provided by the GitLab worker. The mirror is created and
+kept up-to-date by the scheduled run of the pipeline in `ostc-manifest-mirror
+repository`_
+
+.. _ostc-manifest-mirror repository: https://git.ostc-eu.org/OSTC/infrastructure/ostc-manifest-mirror
+
+When the mirror is out-of-date additional git revisions needed to construct the
+workspace are fetched form the respective upstream repositories. Some of the git
+repositories described by the manifest are rather large and are hosted on
+infrastructure outside of the OSTC cloud provider, making this an important
+optimization.
+
+The mirror is automatically published in https://cache.ostc-eu.org/git-repo-mirrors/ostc-develop
+
+Bitbake downloads and sstate-cache
+..................................
+
+The ``.workspace`` job configures bitbake to use a persistent directory that is
+shared between CI jobs, for the location of the ``download`` directory as well
+as the ``sstate-cache`` directory.
+
+The job is using GitLab runner tags to schedule jobs in the environment where
+that shared storage is available. When a new dependency is added or when the
+layers and recipes are changed or updated, the download is automatically
+populated with the necessary source archives. Similarly ``sstate-cache`` is
+populated by all the build jobs present throughout the CI system.
+
+Due to legal restrictions, the caches are split into two pairs, public and private.
+The public cache is automatically published in https://cache.ostc-eu.org/bitbake/
+The private cache, which is used by default, is available on the same volume but it is
+not shared anywhere.
+
+In case the cache is fed with a software package that is, in retrospective
+somehow problematic, for example, by not being freely redistributable, the cache
+can be purged at will.
+
+For details on how cache selection and bitbake configuration looks like, please
+refer to the pipeline source code.

ci/meta-ohos.rst

+.. SPDX-FileCopyrightText: Huawei Inc.
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+The meta-ohos Repository
+========================
+
+.. _meta-ohos repository: https://git.ostc-eu.org/OSTC/OHOS/meta-ohos.git
+
+The `meta-ohos repository`_ contains meta-layers specific to OpenHarmony and is
+the second most important repository after the manifest repository.
+
+The CI pipeline is defined in the file ``.ostc-ci/gitlab-ci.yml``.
+
+Shared Job Definitions
+----------------------
+
+The *meta-ohos* repository does not maintain the list of configurations to
+build. That list is included from the *manifest* repository. In result both
+repositories observe the same set of builds jobs, covering all the supported
+configurations.
+
+The pipeline customizes the ``.build`` job to allow the build process to take
+into account any changes being introduced to the `meta-ohos` repository by the
+incoming pull request. This is done by setting the ``OHOS_CI_GIT_REPO_PATH``
+variable, which is used by the ``.workspace`` job defined in the *manifest*
+repository.
+
+Special Jobs
+------------
+
+build-docs
+..........
+
+This job runs whenever a merge request is made, or when changes land in the
+default branch and in addition, the ``docs/`` directory is modified. This job
+builds the documentation with sphinx and ensures it can be built without any
+warnings or errors.
+
+update-docs
+...........
+
+This job runs whenever changes land on the default branch and affect either the
+pipeline or the ``docs/`` directory. This job *triggers* the pipeline of the
+``OSTC/OHOS/docs`` repository, ensuring that any change to documentation
+present in *meta-ohos* is reflected in the aggregated documentation build
+maintained in the *docs* repository.

ci/xts-acts.rst

+.. SPDX-FileCopyrightText: Huawei Inc.
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+The xts_acts Repository
+=======================
+
+.. _xts_acts repository: https://git.ostc-eu.org/OSTC/OHOS/components/staging/xts_acts.git
+
+The `xts_acts repository`_ contains source code for the OpenHarmony Application
+Compatibility Test Suite. It is relevant to CI because changes proposed here
+result in booting a test image, running ACTS binaries and storing the results
+back into GitLab.
+
+The CI pipeline is defined in the file ``.ostc-ci/gitlab-ci.yml``.
+
+Shared Job Definitions
+----------------------
+
+The *xts_acts* repository does not maintain the list of configurations to
+build. That list is included from the *manifest* repository. In result both
+repositories observe the same set of builds jobs, covering all the supported
+configurations.
+
+The pipeline customizes the ``.build`` job to allow the build process to take
+into account any changes being introduced to the *xts_acts* repository by the
+incoming pull request. This is done by setting ``OHOS_CI_DEVTOOL_RECIPE_NAME``
+and ``OHOS_CI_DEVTOOL_LAYER_PATH`` to effectively use ``devtool`` to upgrade the
+``ohos-xts-acts`` recipe to the source code contained in the proposed change.
+The two variables are used by the ``.workspace`` job defined in the *manifest*
+repository.
+
+The ``.build`` job is further customized to use `wic` to create a bootable
+image. The image is available as an artifact used by a pair of jobs in
+conjunction with ``spread``.
+
+In addition, the pipeline disables all the build jobs with the exception of the
+``linux-qemu-x86_64``. Currently ACTS is only executed on a virtual machine
+operated by ``qemu-system-x86_64``, other builds would only waste resources.
+
+Special Jobs
+------------
+
+This pipeline contains two additional jobs, both related to using ``spread`` to
+boot the built system image and execute ACTS binaries. The jobs are identical
+with the exception of the *spread suite* used.
+
+spread-linux-failing
+....................
+
+This job depends on the ``linux-qemu-x86_64`` job and uses the provided system
+image artifact. The job runs ``spread`` that in turn uses ``qemu-system-x86_64``
+to boot the test image and execute *spread tasks* corresponding to individual
+ACTS programs.
+
+This job runs the *spread suite* ``tests/acts-failing`` and is allowed to fail
+without blocking the pipeline. Tests executed here contain one or more failures
+when working on Linux. Over time, as tests are debugged and ported to Linux,
+they are moved to the ``tests/acts-passing`` test suite.
+
+.. _spread documentation: https://github.com/snapcore/spread
+
+You can refer to upstream `spread documentation`_ for details.
+
+spread-linux-passing
+....................
+
+This job is identical to ``spread-linux-failing`` except that it is not allowed to fail
+and that it runs `tests/acts-passing` spread test suite.
+
+Implementation Highlights
+-------------------------
+
+The system image is constructed with ``wic`` and a customized kick-start file. The
+test image uses a fixed user name and password of ``root`` and ``ohos``
+respectively and relies on network connectivity and ssh support to drive the
+test process.
+
+.. _oh-spread fork: https://git.ostc-eu.org/OSTC/tools/oh-spread
+
+The version of *spread* used here is the `oh-spread fork`_, with additional
+patches applied against the upstream version.

index.rst

@@ -30,3 +30,4 @@ headphones, and AR/VR glasses.
    contributing/README
    ip-policy_implementation-guidelines/index
    ip-policy_policy/index
+   ci/index