Prototype twister testing in LAVA

The goal is to use artifacts from Oniro zephyr builds with twister's --test-only option and let twister do all the heavy-lifting in terms of flashing all the images to the board and getting the results back to LAVA.

try doing it on the local host. Try with philosopher image or one of the zephyr kernel tests once the build for those is fixed if it's working, move this to lava. We can utilize LAVA docker job for this. The only unknown for me atm is how to parse the results from twister but imo shouldn't be too hard.

added to epic &14 (closed)

changed the description

assigned to @sradakovi

set weight to 5

set weight to 7

added teamtesting label

changed milestone to %goofy alpha

changed iteration to Oniro Core Iterations Mar 1, 2022 - Mar 31, 2022

Per the discussion with @stevan.radakovic, this is doable in LAVA by putting the twister test environment inside docker container. Apart from that, we also need build system to provide the test images with the original twister out hierarchy.

Example with kernel/common test suite.

twister-out
├── 96b_nitrogen
│   └── tests
│       └── kernel
│           └── common
│               ├── kernel.common
│               │   └── zephyr
│               │       └── zephyr.elf
│               ├── kernel.common.misra
│               │   └── zephyr
│               │       └── zephyr.elf
│               ├── kernel.common.nano32
│               │   └── zephyr
│               │       └── zephyr.elf
│               └── kernel.common.nano64
│                   └── zephyr
│                       └── zephyr.elf
└── twister.csv

We plan to prepare the docker image required in LAVA, and use zephyr upstream source to build and upload twister output like the above to file server for POC in LAVA. @idlethread, I guess, eventually, we will need the help from build team to build artifacts required by twister for oniro using the bitbake build system.

The obvious benefits with twister are:

• With twister, we can run all the tests in one LAVA job. Currently, we need a separate LAVA job for each test image.
• Twitwister.csv is generated by scripts/twister -p 96b_nitrogen --build-only. Example:

$ cat twister-out/twister.csv 
test,arch,platform,status,extra_args,handler,handler_time,ram_size,rom_size
tests/kernel/common/kernel.common,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.tls,arm,96b_nitrogen,skipped,,na,,0,0
tests/kernel/common/kernel.common.misra,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.nano32,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.nano64,arm,96b_nitrogen,passed,,na,,0,0

Before test image flashing, the scripts/twister --device-testing --hardware-map map.yml -p 96b_nitrogen --test-only command needs to load the csv file to generate a list of tests for the specific platform. Refer to https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/twister#L1113ster also can generate consolidated report in xml format which can be easily converted to LAVA results.

• Tester/developer can use twister on their local environment without LAVA. Currently, we need the help from LAVA dispatcher to parse test output for generating results.

=== Comments from Andrei ===

I'd need to understand more now twister interacts with the build's artefacts. Especially how that csv file is composed and what it includes. The zephyr elf files should already be provided by the build.

==== ends ===

twister.csv is generated by scripts/twister -p 96b_nitrogen --build-only. Example:

$ cat twister-out/twister.csv 
test,arch,platform,status,extra_args,handler,handler_time,ram_size,rom_size
tests/kernel/common/kernel.common,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.tls,arm,96b_nitrogen,skipped,,na,,0,0
tests/kernel/common/kernel.common.misra,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.nano32,arm,96b_nitrogen,passed,,na,,0,0
tests/kernel/common/kernel.common.nano64,arm,96b_nitrogen,passed,,na,,0,0

Before test image flashing, the scripts/twister --device-testing --hardware-map map.yml -p 96b_nitrogen --test-only command needs to load the csv file to generate a list of tests for the specific platform. Refer to https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/twister#L1113

assigned to @chaseqi

Prototype using twister in LAVA on nitrogen device is done. Job example:

device_type: nitrogen
job_name: nitrogen docker test shell - twister kernel tests
timeouts:
  job:
    minutes: 150
  action:
    minutes: 5
  actions:
    wait-usb-device:
      seconds: 40
priority: medium
visibility: public

actions:
- test:
    timeout:
        minutes: 100
    docker:
        image: chaseqi/zephyr-twister:v0.5
        local: true
    definitions:
    - from: inline
      repository:
        metadata:
          format: Lava-Test Test Definition 1.0
          name: debug
        run:
          steps:
          - |
            # check device existence
            pyocd list

            # config test
            cd ~/zephyrproject/zephyr
            wget http://192.168.50.33:8888/nitrogen/twister/map.yml
            cat map.yml
            wget http://192.168.50.33:8888/nitrogen/twister/pyocd_reset.sh
            chmod +x pyocd_reset.sh
            cat pyocd_reset.sh

            # download build images.
            rm -rf twister-out
            wget -qO- http://192.168.50.33:8888/nitrogen/twister/twister-out-fifo.tar.gz | tar xz

            # run test
            source ./zephyr-env.sh
            ./scripts/twister -vvv --device-testing --hardware-map map.yml -p 96b_nitrogen --test-only -T tests/kernel/
      name: debug
      path: inline/debug.yaml

The current problem I am facing here is that twister has no support for arduino devices nano33 ble device. Here is the supported vendor and runner list https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/pylib/twister/twisterlib.py#L4218

With customized west flash argument scripts/twister -vvv --device-testing --device-serial /dev/ttyACM0 -p arduino_nano_33_ble --west-flash="--bossac=/etc/lava-dispatcher-host/build/boards/nano33ble/bossac" --test-only -O twister-out -T tests/kernel/pending/, the board can be flashed, but fails to run test. I am checking twister source code to understand the issue.

Twister connect serial first and then flash test image. On the nano33 board, we need to press reset twice to enter bootloader for flashing, this will disconnect the serial connection. I needed to patch twister to allow to connect after flash.

Here are the LAVA job examples using twister for zephyr testing.

nitrogen: https://lava.ostc-eu.org/scheduler/job/10150/definition

For nitrogen, the test artifacts are generated by command scripts/twister -p 96b_nitrogen --build-only -T tests/kernel/common/

arduino nano33 ble: https://lava.ostc-eu.org/scheduler/job/10161

For the nano33 board, the test artifacts are generated by command scripts/twister -p arduino_nano_33_ble --build-only -T tests/kernel/context. Since the required ttyACM0 is not enabled from the upstream either, I replaced the zephyr.bin file with the context.bin that provided by @brgl.

CC @sradakovi @idlethread @agherzan @pidge

From my point of view, the above LAVA job examples proved that we can use twister in LAVA for zephyr testing on both nitrogen and nano33 ble. IMHO, the remaining dependencies on build team are:

• Enable ttyACM0 for arduino nano33 ble board
• Provide a way to build test artifacts using oniro build system. The build command I used is scripts/twister -p <platform> --build-only tests/kernel. Build recipe/system can just run the command to generate the raw 'twister-out' directory, CI can decide which files should be published.

Nice, thorough work @chaseqi , thanks! I'll consider this ticket close and we can move on with the implementation while keeping the dependencies in mind. I'll open a follow up issue for the next sprint.

added Doing label

closed

removed Doing label

mentioned in issue #21 (closed)

mentioned in merge request !7 (merged)