Random test failures on nitrogen
We started seeing new random failures on nitrogen. I tested with more jobs, most of the jobs reported some random test failures, typically on the -02
device. It used to work fine. Random issue are not easy to fix. I am listing a few directions that came to my mind for the following investigation.
- Device related. Most of the failures are caused by flash error with twister. https://lava.ostc-eu.org/scheduler/job/25681#L536
- It might be also related to the loading on the pi worker. Two new devices added on the same worker wrc-pi4-1.warsaw. When jobs run on both rpi4 and nitrogen devices in parallel, the loading on cpu is pretty high. Once jobs are added on qemu-aarch64-01, I think it will be even worse.
- Timing. We can increase twister internal flash/read timeout to check if helps.
- Test images related. We can test with upstream main branch for comparison.