Research and document use-cases of the Oniro upgrade delta report tool

added to epic &8

changed title from Research and document use-cases to Research and document use-cases of the Oniro upgrade delta report tool

changed milestone to %goofy alpha

set weight to 3

added teamsecurity label

changed epic to &100

changed milestone to %goofy beta

changed iteration to Oniro Core Iterations Jul 1, 2022 - Jul 31, 2022

added Doing label

assigned to @mrybczyn

Upgrade tool use-cases

Starting state

Device vendor DV has an old revision of Oniro and it's layers.
DV has some own layer (not in the default Oniro configuration).
In DV's layers there are some .bbappends files that appends to Oniro's recipes.
DV has specific conf files to build their image. DV has not changed (directly) in any way Oniro's layers.

What can be updated automatically

Oniro's layers

What can be updated with user help

DV append files to match PV in case of a version update of the package

What will be reported after the update

Updated packages
DV's .bbappend status (updated / dropped / unchanged)
CVE status (removed / added / current list)

Ending state

DV has the selected version of Oniro and is able to build with minimal user intervention. DV retains all the custom work done on top of Oniro.

The upgrade tool should have a --dry-run option that is the default (principle of least surprise)
There should be instructions to revert back changes to a local copy of the project (Yes, I'm worried about vendors not having Yocto expertise)
How do you plan to detect the DV layers? What about other upstream Yocto layers that are not present in our base image but used by DV?
Another thing that should be reported is supportability in case we've dropped support for certain packages that the DV uses.

Detect the DV layers

In case we want to create a fresh build automatically

We can compare (before any updates are made) conf/bblayers.conf with oniro/flavours/?/bblayers.conf.sample. Obviously we need to get a parsed version of the content of BBLAYERS and compare extract the new layers that are not in the sample version, then we can just add them to the updated version of bblayers.conf.

In case we want to use a previous build

We can just get the diff of oniro/flavours/?/bblayers.conf.sample and add the new layers to conf/bblayers.conf (while taking into account the specific complete path).

I think all the layers that are not brought with Oniro are to treated equally, we don't care if they are Yocto or not because we're not officially supporting them.

We need to consider also that conf can change and we may end up after the repo sync with old configs that need to be refreshed. This caused an issue when going from v2.0.0-alpha to the latest revision 06f8baf6

Another possible issue we can incur in is when DV has integrated in their build a Yocto layer that in the old revision was not officially supported by Oniro but after the update it is (example meta-java in v2.0.0-alpha2 was not supported but now it is).

In case the DV has NOT committed anything locally in the this specific layer we could take the most recent commit (but in this case ABI could be broken and the build might fail).
In case the DV has committed something in the layer we could let the DV know that the layer folder has been backed up in a specific location and that now we have the supported version of it.

Better solutions need to be found to ensure that the update generates the least amount of build problems and work for the DV.

Open question: Do we support use-made changes to Oniro layers (and the layers we bring in)? I would say we won't, we should ask them to do their own layer(s) with appends on top. This needs to be documented however.

I wouldn't support any direct change to Oniro layers.

There might be some initial pain as vendors learn to work with this, but it'll be worth it from a ongoing maintenance perspective for both the DV and Oniro.

The changes to Oniro should come as a separate layer. If the changes needed are not possible with that approach, we should first understand the limitation - most of the time it is an issue on the upstream (in the case of a vendor we would be the upstream).

assigned to @tony3oo3

buildhistory yocto class

Buildhistory was used before and after a dummy update (v2.0.0-alpha -> 06f8baf6).

What does buildhistory do? From buildhistory docs

When you enable build history, it records information about the contents of each package and image and then commits that information to a local Git repository where you can examine the information.

Is the complete report useful for our use case? No, it's too detailed to be readable after a big update
Does it affect build times? Yes but in a very small amount
Do we need user intervention to use it? No we just need to append 2 lines in local.conf

To get a useful report you need to run the buildhistory-diff command. The first part will print something like this:

Changes to images/qemux86_64/musl/oniro-image-base (files-in-image.txt):
  /usr/lib/libgio-2.0.so.0 changed symlink target from libgio-2.0.so.0.7200.1 to libgio-2.0.so.0.7200.3
  /usr/lib/libglib-2.0.so.0 changed symlink target from libglib-2.0.so.0.7200.1 to libglib-2.0.so.0.7200.3
  /usr/lib/libgmodule-2.0.so.0 changed symlink target from libgmodule-2.0.so.0.7200.1 to libgmodule-2.0.so.0.7200.3
  /usr/lib/libgobject-2.0.so.0 changed symlink target from libgobject-2.0.so.0.7200.1 to libgobject-2.0.so.0.7200.3
  /usr/lib/libgthread-2.0.so.0 changed symlink target from libgthread-2.0.so.0.7200.1 to libgthread-2.0.so.0.7200.3
  /usr/lib/libgio-2.0.so.0.7200.1 moved to /usr/lib/libgio-2.0.so.0.7200.3
  /usr/lib/libglib-2.0.so.0.7200.1 moved to /usr/lib/libglib-2.0.so.0.7200.3
  /usr/lib/libgmodule-2.0.so.0.7200.1 moved to /usr/lib/libgmodule-2.0.so.0.7200.3
  /usr/lib/libgobject-2.0.so.0.7200.1 moved to /usr/lib/libgobject-2.0.so.0.7200.3
  /usr/lib/libgthread-2.0.so.0.7200.1 moved to /usr/lib/libgthread-2.0.so.0.7200.3
  /etc/libnl was added
  /etc/libnl/classid was added
  /etc/libnl/pktloc was added
  /etc/pam.d/sudo was added
  /etc/sudo.conf was added
  /etc/sudoers was added
  /etc/sudoers.d was added
  /etc/sudoers.dist was added
  /etc/sudoers.d/oniro was added
  /etc/sudo_logsrvd.conf was added
  /home/oniro was added
  /home/oniro/.bashrc was added
  /home/oniro/.profile was added
  /usr/bin/cvtsudoers was added
  /usr/bin/sudo was added
  /usr/bin/sudoedit was added
  /usr/bin/sudoreplay was added
  /usr/lib/libnl-3.so.200.26.0 was added
  /usr/lib/libnl-3.so.200 was added
  /usr/lib/libnl-genl-3.so.200.26.0 was added
  /usr/lib/libnl-genl-3.so.200 was added
  /usr/lib/sudo was added
  /usr/lib/sudo/audit_json.so was added
  /usr/lib/sudo/group_file.so was added
  /usr/lib/sudo/libsudo_util.so.0.0.0 was added
  /usr/lib/sudo/libsudo_util.so.0 was added
  /usr/lib/sudo/sample_approval.so was added
  /usr/lib/sudo/sudoers.so was added
  /usr/lib/sudo/sudo_intercept.so was added
  /usr/lib/sudo/sudo_noexec.so was added
  /usr/lib/sudo/system_group.so was added
  /usr/lib/tmpfiles.d/sudo.conf was added
  /usr/sbin/sudo_logsrvd was added
  /usr/sbin/sudo_sendlog was added
  /usr/sbin/visudo was added
  /var/lib/sudo was added
  /var/lib/sudo/lectured was added
  /etc/default/postinst was removed
images/qemux86_64/musl/oniro-image-base: Changes to /etc/group:
  --- /etc/group
  +++ /etc/group
  @@ -46,4 +46,5 @@
   systemd-bus-proxy:x:997:
   avahi:x:998:
   messagebus:x:999:
  +oniro:x:1000:
   nogroup:x:65534:
  --
images/qemux86_64/musl/oniro-image-base: Changes to /etc/passwd:
  --- /etc/passwd
  +++ /etc/passwd
  @@ -21,4 +21,5 @@
   systemd-bus-proxy:x:997:997::/:/sbin/nologin
   avahi:x:998:998::/run/avahi-daemon:/bin/false
   messagebus:x:999:999::/var/lib/dbus:/bin/false
  +oniro:x:1000:1000::/home/oniro:/bin/sh
   nobody:x:65534:65534:nobody:/nonexistent:/sbin/nologin
  --
Changes to images/qemux86_64/musl/oniro-image-base (installed-package-names.txt):
  sudo-lib was added
  sudo was added
  libnl-genl-3-200 was added
  sudo-sudo was added
  libnl-3-200 was added

changed the description

We have quite many requirements here already, time to go for a MVP implementation

closed

removed Doing label

mentioned in epic &100

mentioned in merge request !329 (merged)

Research and document use-cases of the Oniro upgrade delta report tool

Value

Description

In scope

Out of Scope

Acceptance Criteria

Applicable Market Segments

Applicable Personas

Child items ...

Activity