device-testing.rst

On-device Testing

Overview

|main_project_name| implements distributed device testing using Linaro Automation and Validation Architecture (LAVA). This architecture creates an environment where you can operate the necessary physical infrastructure responsible for testing development on real devices, like operating system boot-loader and kernel development, while sharing access to a project-specific software infrastructure used in the public cloud.

How does the CI system work?

The system automatically performs a set of test jobs upon a new or modified pull request. Failed jobs stop the pipeline, allowing you to review build logs, reproduce and resolve the failure locally. The central system maintains a queue and schedules build and test jobs for the available workers. The workers may be auto-scaled, for example, virtual machines in the public cloud, or fixed, for example, a set of physical machines prepared for automated deployment and testing.

Testing Infrastructure

The testing infrastructure consists of a pool of devices physically located at a specific site. These devices are operated and maintained by partner companies and/or individuals. The device maintenance may involve resolving networking problems, swapping out a faulty storage medium, or configuring the device for initial provisioning to the pool.

A site may operate as little as one device or as many as several dozen or hundred, depending on the test suites load and available resources. Sites can thus range from a single desk with a single device, a small rack with several devices in a corporate office, up to a dedicated testing lab with a large number of diverse devices.

Each site is added to the central infrastructure by registering a software-specific service operating on-site and connecting it to the central system. There are two possible site configurations, depending on connectivity to the public cloud:

• If the network connection is poor, building and downloading the images locally is suggested.
• If the network connection is robust, use the central build system directly to build the images, where scalability is easy.

Typically a micro-site that has limited throughput will be bound by the limited number of test devices and will be able to perform the builds locally much faster than being able to pull each new large image from the central system. Typically micro-sites will also see a more limited usage, for example, to support a bootstrap of a new project or preparing the process for automation for a new device.

What does LAVA do?

1. The LAVA job definition template populates the required variables for executing the test job using the values from the Gitlab CI.
2. This job is submitted via REST API, and the LAVA executes the job.
3. Using the callback system, LAVA triggers the last CI job in the loop, the report job.
4. The report job calls LAVA REST API to collect the result.
5. The report is submitted back to Gitlab, and the developers can see the report in their merge request.

References

• For more details on LAVA, see Introduction to LAVA.
• For configuring and adding a new device, see Adding your first devices.
• For more details on LAVA test job definition template, see lava-test.
• For sample test job definition, see sample jobs.
• For our CI test job definition, see CI jobs.
• For LAVA supported device list, see Supported Devices.