diff --git a/ci/docs.rst b/ci/docs.rst new file mode 100644 index 0000000000000000000000000000000000000000..09aa8a2de615554227e7e80dcb9aa683c20c98c3 --- /dev/null +++ b/ci/docs.rst @@ -0,0 +1,53 @@ +.. SPDX-FileCopyrightText: Huawei Inc. +.. +.. SPDX-License-Identifier: CC-BY-4.0 + +The docs Repository +=================== + +.. _docs repository: https://git.ostc-eu.org/OSTC/OHOS/docs.git + +The `docs repository`_ contains the general documentation of the OpenHarmony +project. The documentation is normally written in reStructuredText, with an +important difference. Unlike in a regular project, the documentation here is not +standalone. Instead, the git repository contains symbolic links that are valid +when an entire workspace is constructed with ``git-repo`` and the `manifest +<https://git.ostc-eu.org/OSTC/OHOS/manifest>`_ repository. + +This complicates the testing process and the deployment process, but allows the +resulting documentation to span multiple repositories, permitting code and text +to be conveniently changed in one go. + +Special Jobs +------------ + +build +..... + +The ``build`` job ensures that the documentation can be built with `sphinx-build` +without any warnings or errors. Apart from the complexity of setting up the workspace +as described above, it is fairly typical. + +deploy +...... + +The ``deploy`` job builds aggregated documentation view that is rendered at +https://openharmony.readthedocs.org/. This job effectively constructs the +workspace as described above and then archives the entire documentation source code, +de-referencing any links that were followed to other repositories, and commits the +updated set of files to helper repository which is observed by readthedocs. + +The helper repository is called `openharmony-readthedocs-aggregated +<https://git.ostc-eu.org/OSTC/infrastructure/openharmony-readthedocs-aggregated>`_. +That repository contains a webhook, configured at the level of the GitLab project, which +asks readthedocs to re-build the documentation. + +Implementation Highlights +------------------------- + +Commit Credentials +.................. + +Commit to the aforementioned aggregated repository is allowed by a personal access token +that is set up in that repository. The value of the token is stored as a protected variable +available to the docs repository. diff --git a/ci/index.rst b/ci/index.rst new file mode 100644 index 0000000000000000000000000000000000000000..13c3494de093f672ad9498a1006bef33455f4436 --- /dev/null +++ b/ci/index.rst @@ -0,0 +1,19 @@ +.. SPDX-FileCopyrightText: Huawei Inc. +.. +.. SPDX-License-Identifier: CC-BY-4.0 + +Continuous Integration +====================== + +OpenHarmony ``git`` repositories hosted on https://git.ostc-eu.org/OSTC use +GitLab pipelines for building and testing changes before they are merged. +Individual pipelines are documented in each repository, but some general or more +important elements are described below. + +.. toctree:: + :maxdepth: 1 + + manifest + meta-ohos + xts-acts + docs diff --git a/ci/manifest.rst b/ci/manifest.rst new file mode 100644 index 0000000000000000000000000000000000000000..d49c5f9fb2e01b8bb498720fbfe993ff63d85153 --- /dev/null +++ b/ci/manifest.rst @@ -0,0 +1,201 @@ +.. SPDX-FileCopyrightText: Huawei Inc. +.. +.. SPDX-License-Identifier: CC-BY-4.0 + +The manifest Repository +======================= + +.. _manifest repository: https://git.ostc-eu.org/OSTC/OHOS/manifest.git + +The `manifest repository`_ contains the "map" to all the other repositories used +by OpenHarmony and is thus important from CI point of view. + +The *manifest* repository also contains job definitions that pipelines defined +in other repositories can include. Those are all stored in the `.ostc-ci` +directory. Internally, in the manifest repository, the same rules are used to +create a git-repo workspace, synchronize all the git repositories, initialize +and configure bitbake and build bitbake recipes corresponding to supported +reference boards. + +Shared Job Definitions +---------------------- + +The following GitLab job definitions are shared with other repositories. They +can be customized at the level of a particular git repository, but in general +constitute the set of supported build configurations. + +linux-qemu-x86 +.............. + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=qemux86``. This job checks that OpenHarmony software can be built for +a basic 32bit x86 virtual machine. + +The cache for this job is publicly available. + +linux-qemu-x86_64 +................. + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=qemux86-64``. This job checks that OpenHarmony software can be built +for a basic 64bit x86 virtual machine. + +The cache for this job is publicly available. + +linux-qemu-arm +.............. + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=qemuarm``. This job checks that OpenHarmony software can be built for +a basic 32bit ARMv7 virtual machine. + +The cache for this job is publicly available. + +linux-qemu-arm64 +................ + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=qemuarm64``. This job checks that OpenHarmony software can be built +for a basic 64bit ARMv8 virtual machine. + +The cache for this job is publicly available. + +linux-seco-intel-b68 +.................... + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=seco-intel-b68``. This job checks that OpenHarmony software can be +built for the SECO B68 development board, which contains an Intel x86_64 SoC. + +The cache for this job is not public, pending legal review of any firmware that +may be included. + +linux-seco-imx8mm-c61 +..................... + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=seco-imx8mm-c61``. This job checks that OpenHarmony software can be +built for the SECO C61 development board, which contains an iMX8 SoC. + +The cache for this job is not public, as it contains proprietary elements that +cannot be redistributed without an agreement with Freescale. + +linux-stm32mp1-av96 +................... + +This job builds ``openharmony-image-base-tests`` using ``FLAVOUR=linux`` and +``MACHINE=stm32mp1-av96``. This job checks that OpenHarmony software can be +built for the 96Boards Avenger development board, which contains an STM32MP157 +SoC. + +The cache for this job is not public, pending legal review of any firmware that +may be included. + +zephyr-qemu-x86 +............... + +This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and +``MACHINE=qemu-x86``. This job checks that Zephyr can be built for a basic 32bit +x86 virtual machine. + +The cache for this job is publicly available. + +zephyr-qemu-cortex-m3 +..................... + +This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and +``MACHINE=qemu-cortex-m3``. This job checks that Zephyr can be built for a basic +32bit ARM micro-controller virtual machine. + +The cache for this job is publicly available. + +zephyr-96b-nitrogen +................... + +This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and +``MACHINE=96b-nitrogen``. This job checks that Zephyr can be built for the +96Boards Nitrogen development board, which contains an nRF52832 SoC. + +The cache for this job is not public, pending legal review of any firmware that +may be included. + +zephyr-96b-avenger +.................. + +This job builds ``zephyr-philosophers`` using ``FLAVOUR=zephyr`` and +``MACHINE=96b-avenger96``. This job checks that Zephyr can be built for the +96Boards Avenger development board cortex-M3 core, embedded into STM32MP157 SoC. + +The cache for this job is not public, pending legal review of any firmware that +may be included. + +freertos-armv5 +.............. + +This job builds ``zephyr-philosophers` using ``FLAVOUR=freertos`` and +``MACHINE=qemuarmv5``. This job checks that FreeRTOS can be built for a basic +32bit ARMv5 micro-controller virtual machine. + +The cache for this job is publicly available. + +Special Jobs +------------ + +linux-glibc-qemu-x86_64 +....................... + +This job extends ``linux-qemu-x86_64`` and differs in the following way. + +This job performs a build with the libc switched to ``glibc``. It only runs on a +schedule that is defined in the GitLab project settings for the `manifest` +repository. In practice it runs daily to check if the Linux favour could be +switched back to ``glibc``, from the default ``musl`` that is used right now. + +Implementation Highlights +------------------------- + +Implementation details are stored directly in the manifest repository, as code +comments. Only specific highlights are listed below. + +Local git-repo Mirror +..................... + +The ``.workspace`` job relies on a git-repo mirror that is mounted into the +execution environment provided by the GitLab worker. The mirror is created and +kept up-to-date by the scheduled run of the pipeline in `ostc-manifest-mirror +repository`_ + +.. _ostc-manifest-mirror repository: https://git.ostc-eu.org/OSTC/infrastructure/ostc-manifest-mirror + +When the mirror is out-of-date additional git revisions needed to construct the +workspace are fetched form the respective upstream repositories. Some of the git +repositories described by the manifest are rather large and are hosted on +infrastructure outside of the OSTC cloud provider, making this an important +optimization. + +The mirror is automatically published in https://cache.ostc-eu.org/git-repo-mirrors/ostc-develop + +Bitbake downloads and sstate-cache +.................................. + +The ``.workspace`` job configures bitbake to use a persistent directory that is +shared between CI jobs, for the location of the ``download`` directory as well +as the ``sstate-cache`` directory. + +The job is using GitLab runner tags to schedule jobs in the environment where +that shared storage is available. When a new dependency is added or when the +layers and recipes are changed or updated, the download is automatically +populated with the necessary source archives. Similarly ``sstate-cache`` is +populated by all the build jobs present throughout the CI system. + +Due to legal restrictions, the caches are split into two pairs, public and private. +The public cache is automatically published in https://cache.ostc-eu.org/bitbake/ +The private cache, which is used by default, is available on the same volume but it is +not shared anywhere. + +In case the cache is fed with a software package that is, in retrospective +somehow problematic, for example, by not being freely redistributable, the cache +can be purged at will. + +For details on how cache selection and bitbake configuration looks like, please +refer to the pipeline source code. diff --git a/ci/meta-ohos.rst b/ci/meta-ohos.rst new file mode 100644 index 0000000000000000000000000000000000000000..356a5556fbdcf7984ec080d607770cd22c3db3a3 --- /dev/null +++ b/ci/meta-ohos.rst @@ -0,0 +1,47 @@ +.. SPDX-FileCopyrightText: Huawei Inc. +.. +.. SPDX-License-Identifier: CC-BY-4.0 + +The meta-ohos Repository +======================== + +.. _meta-ohos repository: https://git.ostc-eu.org/OSTC/OHOS/meta-ohos.git + +The `meta-ohos repository`_ contains meta-layers specific to OpenHarmony and is +the second most important repository after the manifest repository. + +The CI pipeline is defined in the file ``.ostc-ci/gitlab-ci.yml``. + +Shared Job Definitions +---------------------- + +The *meta-ohos* repository does not maintain the list of configurations to +build. That list is included from the *manifest* repository. In result both +repositories observe the same set of builds jobs, covering all the supported +configurations. + +The pipeline customizes the ``.build`` job to allow the build process to take +into account any changes being introduced to the `meta-ohos` repository by the +incoming pull request. This is done by setting the ``OHOS_CI_GIT_REPO_PATH`` +variable, which is used by the ``.workspace`` job defined in the *manifest* +repository. + +Special Jobs +------------ + +build-docs +.......... + +This job runs whenever a merge request is made, or when changes land in the +default branch and in addition, the ``docs/`` directory is modified. This job +builds the documentation with sphinx and ensures it can be built without any +warnings or errors. + +update-docs +........... + +This job runs whenever changes land on the default branch and affect either the +pipeline or the ``docs/`` directory. This job *triggers* the pipeline of the +``OSTC/OHOS/docs`` repository, ensuring that any change to documentation +present in *meta-ohos* is reflected in the aggregated documentation build +maintained in the *docs* repository. diff --git a/ci/xts-acts.rst b/ci/xts-acts.rst new file mode 100644 index 0000000000000000000000000000000000000000..66c1fa55771ea625564e9d876ae2ccf04627c375 --- /dev/null +++ b/ci/xts-acts.rst @@ -0,0 +1,82 @@ +.. SPDX-FileCopyrightText: Huawei Inc. +.. +.. SPDX-License-Identifier: CC-BY-4.0 + +The xts_acts Repository +======================= + +.. _xts_acts repository: https://git.ostc-eu.org/OSTC/OHOS/components/staging/xts_acts.git + +The `xts_acts repository`_ contains source code for the OpenHarmony Application +Compatibility Test Suite. It is relevant to CI because changes proposed here +result in booting a test image, running ACTS binaries and storing the results +back into GitLab. + +The CI pipeline is defined in the file ``.ostc-ci/gitlab-ci.yml``. + +Shared Job Definitions +---------------------- + +The *xts_acts* repository does not maintain the list of configurations to +build. That list is included from the *manifest* repository. In result both +repositories observe the same set of builds jobs, covering all the supported +configurations. + +The pipeline customizes the ``.build`` job to allow the build process to take +into account any changes being introduced to the *xts_acts* repository by the +incoming pull request. This is done by setting ``OHOS_CI_DEVTOOL_RECIPE_NAME`` +and ``OHOS_CI_DEVTOOL_LAYER_PATH`` to effectively use ``devtool`` to upgrade the +``ohos-xts-acts`` recipe to the source code contained in the proposed change. +The two variables are used by the ``.workspace`` job defined in the *manifest* +repository. + +The ``.build`` job is further customized to use `wic` to create a bootable +image. The image is available as an artifact used by a pair of jobs in +conjunction with ``spread``. + +In addition, the pipeline disables all the build jobs with the exception of the +``linux-qemu-x86_64``. Currently ACTS is only executed on a virtual machine +operated by ``qemu-system-x86_64``, other builds would only waste resources. + +Special Jobs +------------ + +This pipeline contains two additional jobs, both related to using ``spread`` to +boot the built system image and execute ACTS binaries. The jobs are identical +with the exception of the *spread suite* used. + +spread-linux-failing +.................... + +This job depends on the ``linux-qemu-x86_64`` job and uses the provided system +image artifact. The job runs ``spread`` that in turn uses ``qemu-system-x86_64`` +to boot the test image and execute *spread tasks* corresponding to individual +ACTS programs. + +This job runs the *spread suite* ``tests/acts-failing`` and is allowed to fail +without blocking the pipeline. Tests executed here contain one or more failures +when working on Linux. Over time, as tests are debugged and ported to Linux, +they are moved to the ``tests/acts-passing`` test suite. + +.. _spread documentation: https://github.com/snapcore/spread + +You can refer to upstream `spread documentation`_ for details. + +spread-linux-passing +.................... + +This job is identical to ``spread-linux-failing`` except that it is not allowed to fail +and that it runs `tests/acts-passing` spread test suite. + +Implementation Highlights +------------------------- + +The system image is constructed with ``wic`` and a customized kick-start file. The +test image uses a fixed user name and password of ``root`` and ``ohos`` +respectively and relies on network connectivity and ssh support to drive the +test process. + +.. _oh-spread fork: https://git.ostc-eu.org/OSTC/tools/oh-spread + +The version of *spread* used here is the `oh-spread fork`_, with additional +patches applied against the upstream version. diff --git a/index.rst b/index.rst index 2dcd5190e37df73cf78db4257570b986f35eb138..4242095d9a9b100067805efe763eda8412563255 100644 --- a/index.rst +++ b/index.rst @@ -30,3 +30,4 @@ headphones, and AR/VR glasses. contributing/README ip-policy_implementation-guidelines/index ip-policy_policy/index + ci/index