EF pipeline migration: ci job eddie-zephir failed failed due to permission denied

Problem with job eddie-zephir while migrating to EF gitlab runner.

https://gitlab.eclipse.org/eclipse/oniro-core/eddie/-/jobs/138725

/scripts-2525-138725/step_script: line 166: /etc/dpkg/dpkg.cfg.d/force-unsafe-io: Permission denied
``

Due to this include in the `.gitlab-ci.yml` file:  

• remote: 'https://gitlab.com/zygoon/zephyr-ci-pipelines/-/raw/db83d176cdb5369c7393db6c85ef8ddac1514124/zephyr.yml'

OKD policy does not allow package installation.

It's a blocker for the MR: https://gitlab.eclipse.org/eclipse/oniro-core/eddie/-/merge_requests/52

assigned to @agherzan and @stefanschmidt

changed title from EF pipeline migration: ci job eddie-zephir failed failed due to to EF pipeline migration: ci job eddie-zephir failed failed due to permission denied

marked this issue as related to eclipse-wg/oniro-wg/products-services-oniro-wg/it-services-oniro-wg/pipelines-architecture-oniro-wg#2 (closed)

We can drop that specific configuration for a performance impact. We would need to test to see what that amounts to.

Is this pipeline executing on a shell executor?

Kubernetes executor only.

If this is not executing inside the docker executor, this will not work. If you have a requirement to use unprivileged user inside the container then the pipeline will not work.

Exactly!

I propose to internalize this pipeline in eddie project. So it means:

• copy and modify https://gitlab.com/zygoon/zephyr-ci-pipelines/-/raw/db83d176cdb5369c7393db6c85ef8ddac1514124/zephyr.yml
• create a new docker image inherit from Ubuntu 22.04, and install all necessary package.

WDYT?

The duplication of that is an obvious reason not to go down that route. I'm a bit confused here actually. What is the issue we have here? Why isn't the container process allowed to change files in its root?

Because it's in /etc and you need to be root for that. Anyway, the include pipeline after that try to install packages which will not be possible due to our internal policy:

    - apt install --no-install-recommends -y
        git cmake ninja-build gperf ccache dfu-util device-tree-compiler wget
        python3-dev python3-pip python3-setuptools python3-tk python3-wheel
        xz-utils file make gcc gcc-multilib g++-multilib libsdl2-dev libmagic1

@heurtemattes can you point me to the policy? I'd like to get familiar with this restriction.

that's just okd! you're not root in the container.

And you pointed to the same solution here: sysota#25 (comment 1024781)

@heurtemattes

TIL: OKD.

OK, so this is about running the process in the container as root mapped to root outside of the container. Is the EF foundation using a rootless podman setup for running containers on the workers? These details are the actual policy that I was looking for. Even so, these aspects are orthogonal. The root user in a container doesn't need to be mapped to the root user outside the container. In other words, I can run a rootless container where the user in the container will be root and be able to change the exact files that make the scope of this issue. Here is an example:

$ id -u
1000
$ podman run -ti ubuntu:22.04 bash
[...]
# id -u
0
# echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/force-unsafe-io
# apt update
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
[...]
Fetched 23.4 MB in 2s (13.2 MB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
# apt install git
[...]
done.

And this should make a lot of sense because at the end of the day we ran a process under a non-root user, that creates files under its home directory with the ownership permissions associated with the same user. So you would expect that the process would have full access to those files. The trick is that there is a user map in place thanks to user namespaces:

# cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

You can easily check this if you run a simple rootless container which does:

# echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/force-unsafe-io

You will find the upper filesystem in podman inspect and using that path, the file that was created. Here is an example:

andrei@hostname:~/.local/share/containers/storage/overlay/6eede37221069735aaaa21a125e4b7791761ae73c707728e592d7249ef322b50/diff$ ls -al etc/dpkg/dpkg.cfg.d/
total 12
drwxr-xr-x 2 andrei andrei 4096 Oct 11 17:03 .
drwxr-xr-x 3 andrei andrei 4096 Oct 11 17:03 ..
-rw-r--r-- 1 andrei andrei   16 Oct 11 17:03 force-unsafe-io
andrei@hostname:~/.local/share/containers/storage/overlay/6eede37221069735aaaa21a125e4b7791761ae73c707728e592d7249ef322b50/diff$ cat etc/dpkg/dpkg.cfg.d/force-unsafe-io
force-unsafe-io

This is what makes the problem here orthogonal and makes me not agree with the general "no-root" umbrella policy unless you understand how subordinate UIDs work. It is not OKD to run as root in a container, it is OKD to avoid running as root mapped to the host's root user.

Regarding sysota#25 (comment 1024781). That is not really the same solution/problem (even though similar because even apt just creates. modifies and removes files). I agree that these packages are easier to maintain as part of the container image - hence agreeing there. Here we are also talking about extra root configuration in the container which makes this different.

@heurtemattes I'm happy to accept contributions to the pipeline I've created, so that it uses a docker container. Ideally the container would pre-install the SDK and all the packages.

@zyga I opened a MR with the suggested changes on the .zephyr job: https://gitlab.com/zygoon/zephyr-ci-pipelines/-/merge_requests/14

hi @mrfrank working on this too :-) ubuntu package is not enough, pip package west should be also considerated.

https://gitlab.com/sebastien.heurtematte1/zephyr-ci-pipelines/-/tree/refactor/zephyr-docker

sorry, I'll leave it to you :)

MR Opened: https://gitlab.com/zygoon/zephyr-ci-pipelines/-/merge_requests/16

Maybe we should discuss about provided this project under EF gitlab with dedicated runner. I had problems with gitlab and their new policies around gitlab shared runner minutes consumption. That why you'll see buildkit configuration in gitlab-ci.yml.

mentioned in merge request !52

mentioned in commit 88f70aae

mentioned in commit f81a4c21

mentioned in commit c876b97e

mentioned in commit 8b788570

mentioned in commit e50db504

mentioned in commit 5bb12f7f

mentioned in commit 2e243154

open an issue about failing CI: https://gitlab.com/zygoon/zephyr-ci-pipelines/-/issues/3

Works fine now! https://gitlab.eclipse.org/eclipse/oniro-core/eddie/-/jobs/181151