device-testing.rst

On-device Testing

Overview

|main_project_name| implements distributed device testing using Linaro Automation and Validation Architecture (LAVA). This architecture creates an environment where you can operate the necessary physical infrastructure responsible for testing development on real devices, like operating system boot-loader and kernel development, while sharing access to a project-specific software infrastructure used in the public cloud.

How does the CI System Work?

The system automatically performs a set of test jobs upon a new or modified merge request. Failed jobs stop the pipeline, allowing you to review build logs, reproduce and resolve the failure locally. The central system maintains a queue and schedules build and test jobs for the available workers. The workers may be auto-scaled, for example, virtual machines in the public cloud, or fixed, for example, a set of physical machines prepared for automated deployment and testing.

Testing Infrastructure

The testing infrastructure consists of a pool of devices physically located at a specific site. These devices are operated and maintained by partner companies and/or individuals. The device maintenance may involve resolving networking problems, swapping out a faulty storage medium, or configuring the device for initial provisioning to the pool.

A site may operate as little as one device or as many as several dozen or hundred, depending on the test suites load and available resources. Sites can thus range from a single desk with a single device, a small rack with several devices in a corporate office, up to a dedicated testing lab with a large number of diverse devices.

Each site is added to the central infrastructure by registering a software-specific service operating on-site and connecting it to the central system. There are two possible site configurations, depending on connectivity to the public cloud:

• If the network connection is poor, building and downloading the images locally is suggested.
• If the network connection is robust, use the central build system directly to build the images, where scalability is easy.

Typically a micro-site that has limited throughput will be bound by the limited number of test devices and will be able to perform the builds locally much faster than being able to pull each new large image from the central system. Typically micro-sites will also see a more limited usage, for example, to support a bootstrap of a new project or preparing the process for automation for a new device.

Oniro CI Flow

Oniro CI Flow

1. New merge request triggers the CI pipeline.
2. The pipeline build stage builds and uploads artifacts.
3. Test job under test stage is triggered once the dependent build job finishes.
4. The LAVA job definition template populates the required variables for executing the test job using the values from the GitLab CI.
5. The test job is submitted using the REST API, and the LAVA executes the job.
6. Report job is added to the pipeline, if all the dependent test jobs are added.
7. After all LAVA test jobs for the pipeline finish, using the callback system, LAVA triggers the last CI job in the loop, the report job.
8. The report job calls LAVA REST API to collect the test results.
9. The report is submitted back to GitLab, and the test report can be seen in the merge request.

Setting up Remote LAVA Worker

A LAVA worker performs test execution in LAVA. LAVA workers can connect to the LAVA server on a local network or using the internet. We called it remote worker as it can be located anywhere that has internet access.

A remote LAVA worker can be deployed manually by following this guide Running the LAVA dispatcher in a Docker container. The process has been automated using ansible as well.

Following are the steps for automatic deployment.

1. Install Debian buster on your worker.

2. Clone LAVA Ansible Playbooks repository.

   git clone https://git.ostc-eu.org/OSTC/infrastructure/lava/lava-playbooks.git

3. Define a group of worker/workers in hosts and related group variables under group_vars.

   hosts example:

   [worker_dev]
   # local VMs for dev
   192.168.56.103 worker_name=<hostname.lab_name> server_token=<worker_token>
   192.168.56.104 worker_name=<hostname.lab_name> server_token=<worker_token>

   Worker token is generated on LAVA server. Ask LAVA administrator to create your worker and share authorization token.

   group_vars/worker_dev.yaml example:

   lava_config_repo: https://git.ostc-eu.org/OSTC/infrastructure/lava/lava-playbooks.git
   lava_config_repo_branch: master
   # sudo user without password
   ansible_user: <username>
   server_url: http://192.168.56.102

4. Run Ansible playbook for worker deployment.

   Deploy to a worker group:

   ansible-playbook -l worker_dev playbooks/deploy-docker-worker.yaml -b

   Deploy to a specific worker:

   ansible-playbook -l 192.168.56.103  playbooks/deploy-docker-worker.yaml -b

5. Check service status with command systemctl status lava-docker-worker.service.

6. You can also optionally provide arbitrary Dockerfile to LAVA docker worker.

Adding a Device to LAVA

Test devices are connected to their local LAVA worker and manged by LAVA server. The following wiki pages have been created for adding devices to LAVA.

• How to deploy wic image on Raspberry Pi in LAVA
• Adding Arduino Nano 33 BLE Board to LAVA Lab

References

• For more details on LAVA, see Introduction to LAVA.
• For configuring and adding a new device, see Adding your first devices.
• For more details on LAVA test job definition template, see lava-test.
• For sample test job definition, see sample jobs.
• For our CI test job definition, see CI jobs.
• For LAVA supported device list, see Supported Devices.