doc: add ADR for using immutable file system images

I wanted to design SystemOTA around squashfs or erofs instead of the
more traditional writable file system or OS-tree like approaches, but
the exact reasons for that are complex and worth capturing.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>

doc/adr/0004-use-immutable-file-system-images.md

+<!--
+SPDX-License-Identifier: CC-BY-4.0
+SPDX-FileCopyrightText: Huawei Inc.
+-->
+# 4. Use immutable file system images
+
+Date: 2021-07-09
+
+## Status
+
+Accepted
+
+## Context
+
+System updates can fundamentally decide between one of two models. At some level
+you either use a regular file system and put files into it or you start with a
+system image and deal with managing mutability or delta against said image.
+
+Each approach has benefits and problems. The general trend is to move away from
+the writable images, especially when scale of deployment grows and when the
+scope narrows. Using read-only images makes it practical to sign and verify the
+images, it makes it practical to verify the integrity of the system, to offer a
+_factory reset_ feature which essentially removes all data and brings the device
+back to its initial state and lastly it's a great starting point for any delta
+updates, as the system is guaranteed to have a well-known image at all times.
+
+There is still wiggle room in using images, as the images can be mounted or
+unpacked. From the point of view of SystemOTA this decision mostly does not
+matter. Internally the OTA service will provide an image and the rest of
+platform integration will make use of the image. There is an unspoken assumption
+that the images _are_ mounted as this is both faster and actually atomic and
+resilient to corruption during partially completed copy process.
+
+## Decision
+
+Use immutable file system images as the underlying mechanism for providing updates.
+
+## Consequences
+
+Immutable images are friendly to delta compression, where a delta image can be
+constructed between any two images. There are off the shelf tools available for
+both generic blobs as well as specific file systems (notably squashdelta), which
+will allow us to reduce development cost.
+
+To be able to keep state, the overall system must be able to combine the
+immutable file system image with a mutable file system kept on another partition
+and define a set of rules as to which specific places are kept persistently,
+which are mutable but ephemeral and which can remain immutable entirely. Given
+the complexity of a typical minimal Linux system, this is still more art than
+science, so there is going to be considerable integration work necessary for a
+practical end-user result.
+
+All sorts of interesting security mechanisms are possible as a consequence of
+this decision. For a specific example the hardware root of trust may accept a
+signed kernel boot loader which is copied to a platform specific location during
+image construction. The boot loader can then load, verify and measure a signed
+kernel image combined with an initrd image. The initrd image can further locate
+and mount the root file system image. Thanks to the fact that groups of systems
+can share the exact same image, verify it against a public key which is
+embedded into the kernel+initrd image. Lastly, the writable file system can be
+mounted in a way which, for example, prevents code execution, what may be
+required for tightly locked down, special-purpose platforms.
+
+As another consequence, the system has well-defined mutable state which can be
+discarded based on certain conditions (e.g. pressing a specific button) providing 
+a general recovery mechanism bringing a device back to the initial state.
+
+One last consequence is that the update system is intrinsically atomic at the
+cost of reboot. Since the image is designed to contain both the root file system
+and the kernel, any change, big or small, is implemented by re-booting into the
+new system. This is appropriate for a *system* update process, where such
+changes are expected, but may be undesired for *application* update process,
+where it may lead to frequent updates and disruptive downtime. It is recommended
+to pair the system update process with a separate mechanism for updating
+applications that exist outside of the system image, as those can, due to their
+limited nature, be performed without a full system restart.