System OTA for Linux Gateways

System OTA is a robust, unattended update system for Linux Gateways, suitable for home automation gateways and other devices where reliability is paramount.

Architecture Overview

The update system is comprised of the following elements:

• Immutable file system used as rootfs with:
  • Provisions for maintaining state of select files and directories.
  • RAUC pre-installed and pre-configured for storage slots.
  • SysOTA pre-installed and pre-configured for the boot loader, update server and package name.
• Shared or dedicated[1] NetOTA server providing update meta-data.
• Shared or dedicated public storage server for storing image blobs.

Shared or dedicated in the sense that either a special purpose installation or a shared installation used by many different device types, operators and manufacturers are both allowed by the system.

Development

This is a somewhat typical Go and make project. You can use plain go to build and test most things. For a more complete experience, including to access the set of integration tests and various static analysis tools, you should install a few more dependencies:

• make
• zmk
• reuse
• spread

Running make will build everything. There's a build-time configuration system. To create the configure script run make configure.

To try the service out in practice you may want to do this though:

make configure
./configure --prefix=/usr
make
sudo make install
sudo systemctl enable sysotad.service

As you iterate you may need to refresh in-memory definition of systemd unit files:

sudo systemctl daemon-reload

To remove what you've installed run the uninstall target:

sudo make uninstall

Remember that uninstall is stateless, if you want to rename a file, first uninstall, then rename and finally install again.

Testing

The project has a combination of unit tests, available through the standard go interface, and integration tests implemented with spread. To run all kinds of tests invoke make check. This will skip and report tests that depend on tools you do not have locally.

Please try not to send patches which regress the test suite.

Go Unit Tests

There's a wide range of standard Go unit tests available. To execute those simply run:

go test ./...

Go Static Analysis

SystemOTA uses a wide array of Go static analysis tools, including the standard go vet as well as the third-party: staticcheck, goconst, errcheck, wsl and ineffassign.

All those checks are integrated into make check and make static-check. The precise goals for each individual tool can be discovered by tab-completing make static-check-go-.

The tools are usually not packaged so you should install them directly. Assuming you are using Go 1.16+ you can do that with with:

go install honnef.co/go/tools/cmd/staticcheck@latest
go install github.com/jgautheron/goconst/cmd/goconst@latest
go install github.com/kisielk/errcheck@latest
go install github.com/bombsimon/wsl/v3/...@latest
go install github.com/gordonklaus/ineffassign@latest

REUSE compliance checks

REUSE is a legal compliance toolchain which, among others, ensures that each file is accompanied by a machine readable copyright and license.

REUSE checks are integrated into make check and make static-check. The precise goal is make static-check-reuse.

The REUSE toolchain can be installed from the system packager (usually something like python3-reuse) or with pip by running:

pip install reuse

Spell checker

SystemOTA uses a cspell based spell checker. The same spell checker, along with the exception database, can be used from Visual Studio Code and from command line. The list of known exceptions is kept in the file cspell.json.

The spell checker is integrated into make check and make static-check. The precise goals make static-cspell-lint.

To install the cspell with npm by running:

npm install --global cspell

Spread system checks

To execute integration tests you will need a working installation of spread and lxd or qemu. Currently Qemu is preferred, as it allows performing mount operations inside the virtual machine. GitLab CI also uses qemu as LXD cannot be used from a docker container. For more information on how to prepare a test environment with qemu, refer to the spread-qemu-hirsute job defined in .gitlab-ci.yml file.

A compatible build of spread is available from the Ubuntu PPA ppa:zyga/oh-tools. It can be also compiled from sources at https://git.ostc-eu.org/OSTC/tools/oh-spread/.

LXD is available as a snap package. Install it with sudo snap install lxd and then initialize it with sudo lxd init --auto.

To execute integration tests run:

make check-spread

If you use spread directly you may want to pass some options to spread, namely -reuse -resend to speed up subsequent runs by keeping the ephemeral container or virtual machine, with all the build dependencies and cache, around.

Packaging

If you are looking at packaging, the typical ./configure --prefix=/usr && make && make install trio should do the trick but for a real installation you will need to configure SystemOTA and RAUC inside your operating system image, at least to select the boot loader and to describe the partition layout to RAUC.

You may want to enable the sysotad.service systemd unit after installation.

Yocto meta-layers

There's an official meta-layer with SystemOTA in the Oniro repository at https://booting.oniroproject.org/distro/oniro/ in the meta-oniro-core layer.