Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • srctracker
  • feat/ef-runner
  • ipk-npm
  • test-bin-imgs
  • srctracker-POC-v1
  • 0.1.0
7 results
Compare
Name Last commit Last update
LICENSES
docs
.gitlab-ci.yml
LICENSE
README.md
aliensrc_creator.py
fetch_configure_sources.py
get_sources_binaries.py
tinfoilhat.py
README.md
SPDX-License-Identifier: GPL-2.0-only
SPDX-FileCopyrightText: 2021 Alberto Pianon <pianon@array.eu>

TinfoilHat

Tinfoil is a wrapper around BitBake's internal code that can be used to get the value that environment variables actually assume at build time.

TinfoilHat is a wrapper around Tinfoil, in order to do Software Composition Analysis (SCA) on bitbake-based projects, by packaging software components and reconstructing the Bill of Material (BoM).

TinfoilHat has been designed to handle cases in which a single Yocto project has multiple hardware targets/images, or in which multiple Yocto projects share a substantial amount of base source components, which however may slightly differ from each other - even if they share the same name, version and revision (typically, when the upstream component remains the same, but the applied downstream patches may differ between different build targets).

In such cases, to avoid source package proliferation (and resulting confusion) and to aggregate metadata to be used for SCA, we need to capture all possible "variants" of the sources used by each recipe for different build targets/images (see docs/recipe_variants.md for more details) and to create a single source package for each recipe aggregating all such possible variants (see Aliensrc Creator below).

TinfoilHat Installation

TinfoilHat must be run on a build machine against one or more bitbake build dirs (the typical use case is the latter - i.e. multiple build dirs - as explained before).

It requires python 3.8 or later and some system dependencies that can be installed with the following command (for debian/ubuntu):

sudo apt install python3-yaml rsync rpm unzip xz-utils lzip git

To install it, just copy tinfoilhat.py to ~/.local/bin or to any other executable path and make it executable, eg:

cp tinfoilhat.py ~/.local/bin/tinfoilhat && chmod +x ~/.local/bin/tinfoilhat

TinfoilHat Usage

Every time you use it, you must first set the PYTHONPATH environment variable to specific bitbake lib dir used to bake the build you need to scan, eg.:

export PYTHONPATH=/build/yocto/sources/poky/bitbake/lib

Please keep in mind that if you need to scan multiple build dirs, they should share the same bitbake version/dir.

IMPORTANT NOTE: if you need to parse build dirs generated with different versions of bitbake, every time you run tinfoilhat on a build dir, you need to set PYTHONPATH to the path of the corresponding bitbake version. Using a mismatching version of bitbake/tinfoil may possibly lead to inconsistent results or errors.

Like Tinfoil, TinfoilHat must be run against one or more bitbake build dirs after build processes successfully ran, to get metadata about those specific build(s).

IMPORTANT NOTE: in order to allow TinfoilHat to work under any circumstances (especially if any bitbake recipes have git shallow cloning enabled), you should set BB_GENERATE_MIRROR_TARBALLS = "1" in conf/local.conf before build.

usage: tinfoilhat [-h] OUTPUT_DIR PROJECT_NAME RELEASE_TAG [BUILD_DIR [BUILD_DIR ...]]

positional arguments:
  OUTPUT_DIR    dir where to save json files with recipe metadata
  PROJECT_NAME  project name (arbitrary)
  RELEASE_TAG   project's release tag (arbitrary)
  BUILD_DIR     Bitbake build dir(s) to parse (wildcards allowed).Build dir(s)
                must have a conf/ subdir inside, with bitbake configuration
                files.

optional arguments:
  -h, --help    show this help message and exit

To run it against one build only:

tinfoilhat v0.1.0-qemux86.BoM/ ohos v0.1.0 /build/yocto/build-linux-qemux86

To run it against multiple builds you can also use wildcards:

tinfoilhat v0.1.0.BoM/ ohos v0.1.0 /build/yocto/build-linux-*

In the first case, the generated BoM files (in json format, one for each recipe) will include recipes and packages from only one build; in the second case, they will include all recipes and packages from all builds (machine and image tags will be attached to recipes/packages, assuming that each build dir is for one machine only, and may contain more than one built image).

TinfoilHat Output Data Structure

Metadata extracted by TinfoilHat generally refer to corresponding bitbake's build metadata, with some caveats.

The main issue is that Yocto's design and recipe versioning policy may lead to many different source variants tagged with the same package name, version and revision, even if they use different sets of source files (eg. same upstream source tarball but different patches, init scripts or config files).

  • on the one hand, we may want to group variants that share the same upstream source and belong to the same project release but are aimed at different target machines/images (so we add tags both at recipe and at source file level to tell which source files are actually used and when, depending on the build target)
  • on the other hand, we need to uniquely tag such variants (or grouped variants), so we add a unique variant hash to bitbake recipe version and revision.

To understand better the theory and the rationale behind it, check the docs

To see how this is implemented in practice, see comments in the example data below.

Click to see a commented example of the json output data structure of TinfoilHat 
{
  "linux-yocto-5.10.21+gitAUTOINC+8c8f6a791b_8c516ced69-r0-11a46782": {
    "recipe": {
      "metadata": {                       # metadata from bitbake/tinfoil

        "name": "linux-yocto",            # name (with possible prefix|suffix
                                          # added by bitbake, eg `-runtime`)

        "base_name": "linux-yocto",       # base_name without any prefix or
                                          # suffix
        "version": "5.10.21+gitAUTOINC+8c8f6a791b_8c516ced69",
        "revision": "r0",
        "variant": "11a46782",            # since (apparently) the same recipe,
                                          # with the same version and revision,
                                          # can include different source files
                                          # depending on bitbake layer overlays,
                                          # yocto project branches, the weather,
                                          # etc. we need to uniquely tag each
                                          # source "variant"; here we use the
                                          # short (first 8 digits) sha1 checksum
                                          # of the hashes of all source files
                                          # (see chk_sum below, after
                                          # source_files)

        "author": null,
        "homepage": "",
        "summary": "Linux kernel",
        "description": "Linux kernel.",
        "license": "GPLv2",               # bitbake license expressions are NOT
                                          # in SPDX format!

        "depends_provides": {             # build deps (including cross-   
                                          # compilation deps), grouped by target
                                          # machine
          "qemuarm64": {
            "depends": ["bc-native","bc-native","bison-native","gmp-native","kmod-native","libgcc","lzop-native","openssl-native","util-linux-native","virtual/aarch64-poky-linux-musl-binutils","virtual/aarch64-poky-linux-musl-gcc","xz-native"],
            "provides": ["linux-yocto","virtual/kernel"]
          },
          "qemux86": {
            "depends": ["bc-native","bc-native","bison-native","elfutils-native","gmp-native","kmod-native","lzop-native","openssl-native","util-linux-native","virtual/i686-poky-linux-musl-binutils","virtual/i686-poky-linux-musl-gcc","xz-native"],
            "provides": ["linux-yocto","virtual/kernel"]
          },
          "qemux86-64": {
            "depends": ["bc-native","bc-native","bison-native","elfutils-native","gmp-native","kmod-native","lzop-native","openssl-native","util-linux-native","virtual/x86_64-poky-linux-musl-binutils","virtual/x86_64-poky-linux-musl-gcc","xz-native"],
            "provides": [ "linux-yocto", "virtual/kernel" ]
          }
        }
      },
      "cve_metadata": {                   # metadata used by cve-check.bbclass,
        "cve_product": [                  # useful if one wants to replicate
          {                               # the same workflow to periodically
            "vendor": null,               # check CVEs. See cve-check.bbclass
            "product": "linux_kernel"     # code and documentation on yocto
          }                               # website for more details.
        ],
        "cve_version": "5.10.21",
        "cve_version_suffix": null,
        "cve_check_whitelist": []
      },
      "source_files": [
        {
          "rootpath": "/build/common/downloads/git2",
          "relpath": "git.yoctoproject.org.linux-yocto.git",
                                          # local directories, needed by
                                          # aliensrc_creator

          "src_uri": "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=v5.10/standard/qemuarm64;",
                                          # src uri, exactly as it appears in
                                          # bitbake SRC_URI env variable

          "sha1_cksum": null,
          "git_sha1": "8c516ced69f41563404ada0bea315a55bcf1df6f",
                                          # in this case, we don't have a
                                          # downloaded source tarball or a local
                                          # source file, but a git repo, so we
                                          # use git commit sha1 to hash the
                                          # source
          "tags": [
            "ohos/v0.1.0/openharmony-linux/qemuarm64"
                                          # it means that this source is used
                                          # only when building for that
                                          # particular machine (qemuarm64) in
                                          # that particular project release
                                          # (ohos v0.1.0) and distro
                                          # (openharmony-linux). See other
                                          # comments on tags below for more
                                          # details
          ]
        },
        {
          "rootpath": "/build/common/downloads/git2",
          "relpath": "git.yoctoproject.org.yocto-kernel-cache",
          "src_uri": "git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=kernel-meta",
          "sha1_cksum": null,
          "git_sha1": "8c8f6a791bed6e8f675236946e805a7fc489e382",
          "tags": []                      # "no tags" here means that this
                                          # source is used in all project
                                          # releases, distros, target machines
                                          # and images listed in recipe's
                                          # general "tags" field (see below)
        },
        {
          "rootpath": "/build/ohos/v0.1.0/sources/poky/../meta-ohos/meta-ohos-core/recipes-kernel/linux/linux/",
          "relpath": "misc.cfg",
          "src_uri": "file://misc.cfg",
          "sha1_cksum": "55b5fd3111b47c4b9d694a9be71571f7c9d0f7fd",
                                          # this is a local file, so we use
          "git_sha1": null,               # sha1 checksum to hash it (see above)
          "tags": []
        },
        {
          "rootpath": "/build/ohos/v0.1.0/sources/poky/../meta-ohos/meta-ohos-acts/recipes-kernel/linux/linux-yocto/",
          "relpath": "0001-fragment-jffs2-tests.cfg",
          "src_uri": "file://0001-fragment-jffs2-tests.cfg",
          "sha1_cksum": "a920c85f59efdb30d68e6e93111bb1b6f6044d21",
          "git_sha1": null,
          "tags": []
        },
        {
          "rootpath": "/build/common/downloads/git2",
          "relpath": "git.yoctoproject.org.linux-yocto.git",
          "src_uri": "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=v5.10/standard/base;",
          "sha1_cksum": null,
          "git_sha1": "8c516ced69f41563404ada0bea315a55bcf1df6f",
          "tags": [
            "ohos/v0.1.0/openharmony-linux/qemux86",
            "ohos/v0.1.0/openharmony-linux/qemux86-64"
          ]
        }
      ],
      "chk_sum": "11a46782872d29db3bfdaf7979f9ff130c20fe77"
                                          # source package checksum calculated
                                          # by picking each source file's
                                          # git_sha1 or sha1_cksum (depending of
                                          # which is available), sorting all of
                                          # them (alphabetically) and
                                          # concatenating them in a string, and
                                          # calculating the sha1 hash of that
                                          # string
    },
    "tags": [
      "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-base",
      "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-base-tests",
      "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra",
      "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra-tests",
      "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-base",
      "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-base-tests",
      "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-extra",
      "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-extra-tests",
      "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-base",
      "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-base-tests",
      "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-extra",
      "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-extra-tests"
    ],
                                          # recipe's general tag list; each tag
                                          # is a path consisting of:
                                          # project/release/distro/machine/image
                                          # and means that this recipe "variant"
                                          # (see above) is used for all the
                                          # listed targets (apart from tagged
                                          # source files, that are used only for
                                          # the targets described by "their"
                                          # tags - see above)

    "packages": {                         # binary packages generated from
                                          # source files
      "kernel-5.10.21-yocto-standard-5.10.21+gitAUTOINC+8c8f6a791b_8c516ced69-r0+ohos_v0.1.0+openharmony-linux+qemuarm64": {
        "package": {
          "metadata": {
            "name": "kernel-5.10.21-yocto-standard",
            "base_name": "kernel-base",
            "version": "5.10.21+gitAUTOINC+8c8f6a791b_8c516ced69",
            "revision": "r0",
            "package_arch": "qemuarm64",
            "recipe_name": "linux-yocto",
            "recipe_version": "5.10.21+gitAUTOINC+8c8f6a791b_8c516ced69",
            "recipe_revision": "r0",
            "license": "GPLv2",
            "summary": "Linux kernel",
            "description": "Linux kernel.",
            "depends": [
              null
            ],
            "provides": [
              "kernel-5.10.21-yocto-standard",
              "kernel-base (=5.10.21+git0+8c8f6a791b_8c516ced69)"
            ]
          },
          "files": {
            "file_dir": "/build/ohos/v0.1.0/linux-qemuarm64/tmp/deploy/rpm/qemuarm64/kernel-5.10.21-yocto-standard-5.10.21+git0+8c8f6a791b_8c516ced69-r0.qemuarm64.rpm",
            "files": [
              {
                "path": "/lib/modules/5.10.21-yocto-standard/modules.builtin",
                "sha256": "d936dc2e589e4889247595cbaffa318e0bd33d6f57c2fcd45b4754f3441265fd",
                                          # sha256 is used here instead of sha1
                                          # for compatibility and consistency
                                          # with rpm command output
                "size": 7349
              },
              {
                "path": "/lib/modules/5.10.21-yocto-standard/modules.builtin.modinfo",
                "sha256": "f5141ad0b03b60684a6c7c9f5ed0dff8756736eeb7f4c3f46c1a59fb75421021",
                "size": 55089
              },
              {
                "path": "/lib/modules/5.10.21-yocto-standard/modules.order",
                "sha256": "200b25cca75d47961204fc7d9de8d4d08dd286dc57f30922bcbab95d0bc643d9",
                "size": 8705
              }
            ]
          },
          "chk_sum": "27980954d3f2dec32ecaa1613801e16593d5f7fa"
                                          # sha1 hash of the sorted and
                                          # concatenated sha256 checksums of all
                                          # binary files
        },
        "tags": [
          "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-base",
          "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-base-tests",
          "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra",
          "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra-tests"
        ]
      },
  # [...] other binary packages here
    }
  }
}

Aliensrc Creator

Aliensrc Creator Installation

Aliensrc Creator, like TinfoilHat, must be run on a build machine against bitbake build dirs.

It requires python 3.8 or later, and git (it uses the git archive command to create source archives in a deterministic way[^1]).

[^1] i.e. to always get the same output archive file with the same checksum when archiving the same revision of a git repo - with tar it would not be possible, in case of different timestamps

To install it, just copy aliensrc_creator.py to ~/.local/bin or to any other executable path and make it executable, eg:

cp aliensrc_creator.py ~/.local/bin/aliensrc_creator && chmod +x ~/.local/bin/aliensrc_creator

Aliensrc Creator Usage

Like Tinfoil and TinfoilHat, Aliensrc Creator must be run against one or more bitbake build dirs after build processes successfully ran, to create source packages from those specific builds.

It takes as input the .tinfoilhat.json files created by TinfoilHat, as well as the downloaded source tarballs, the local files and the git repos cloned within bitbake build dirs.

While archives and files are copied as such, git repos are copied in a temporary directory, checked out against the commit indicated in bitbake's SRCREV env variable, and then compressed (via git archive) in a .tar.xz archive.1

usage: aliensrc_creator [-h] [--tmpdir TMPDIR] [--skip-existing] OUTPUT_DIR [TINFOILHAT_DIR [TINFOILHAT_DIR ...]]

positional arguments:
  OUTPUT_DIR       dir where to save aliensrc packages
  TINFOILHAT_DIR   dir(s) where to look for .tinfoilhat.json files (wildcards allowed)

optional arguments:
  -h, --help       show this help message and exit
  --tmpdir TMPDIR  temporary dir to use for package creation
  --skip-existing  skip already existing packages

Since aliensrc_creator uses multiprocessing it can create multiple packages in parallel. This may lead to high disk usage for temporary files. To avoid disk full errors if your /tmp dir does not have enough space, you may choose a custom temporary dir through the --tmpdir argument.

Aliensrc Creator Output Data Structure

Here is an example of the structure of an alien package named zlib-1.2.11-r0.aliensrc. The file-extension .aliensrc is mandatory, the name is arbitrary. An alien package is a tarball with no compression. It must have an internal structure like the following:

.
├── aliensrc.json
└── files
    ├── libdrm-2.4.101.tar.xz
    └── _tagged
        └── ohos
            └── v0.1.0
                └── openharmony-linux
                    └── stm32mp1-av96
                        ├── 0001-tests-modetest-automatic-configuration.patch
                        ├── 0002-tests-util-smtpe-increase-alpha-to-middle-band.patch
                        ├── 0003-tests-modetest-set-property-in-atomic-mode.patch
                        └── 0004-tests-modetest-close-crtc.patch

The file aliensrc.json is mandatory; it should be added first, at the beginning of the tarball file (so it can be extracted faster) and contains all metadata information of this alien package.

Source archives or files that are used in any target machine/image are simply put in the files directory, possibly within subdirs. The meaning of such subdir names is context dependent (see below).

Context-dependent subdir names and metadata

For source packages generated from bitbake build dirs by tinfoilhat, source archives or files that are used only for one or more specific targets are put in a separate _tagged/<project>/<release>/<distro>/<machine>/<image>/ subdir. If a file is used in all images of a specific machine, the subdir would be _tagged/<project>/<release>/<distro>/<machine>/; and so on. Such subdir path reflects the corresponding tags in aliensrc.json metadata (see the example data above - for tinfoilhat - and below - for aliensrc).

click to see aliensrc.json data structure example 
{
    "version": 1,                   # the version of this json spec

    "source_package": {             # the data part of this source package

        "name": [                   # some packages have more than one
            "libdrm"                # name/alias, ordered by priority
        ],                          #(top=most important)
        "version": "2.4.101-r0",
        "manager": "bitbake",       # the build system from where we extracted
                                    # this source package

        "metadata": {               # any metadata (tipically, metadata
                                    # extracted from the build system). This
                                    # structure is not defined, nor mandatory.
            "name": "libdrm",       # For the meaning of these specific metadata
            "base_name": "libdrm",  # fields, see TinfoilHat documentation
            "version": "2.4.101",   # above
            "revision": "r0",
            "variant": "50dbf0c6",
            "author": null,
            "homepage": "http://dri.freedesktop.org",
            "summary": "Userspace interface to the kernel DRM services",
            "description": "The runtime library for accessing the kernel DRM services.  DRM stands for \\\"Direct Rendering Manager\\\", which is the kernel portion of the \\\"Direct Rendering Infrastructure\\\" (DRI).  DRI is required for many hardware accelerated OpenGL drivers.",
            "license": "MIT"
        },
        "files": [                  # files, that are included in the "files"
                                    # folder inside the alien package
            {
                "name": "libdrm-2.4.101.tar.xz",
                "git_sha1": null,
                "sha1_cksum": "c08ba0e9b19cea7cd2bbd9fcc39218ac443b41a5",
                                    # since this is a downloaded file, we use
                                    # sha1 checksum to hash it. If it were a git
                                    # repo, we would use both the sha1 of the
                                    # git commit, and the sha1 checksum of the
                                    # tarball archive generated from the git
                                    # repo (checked out at that specific commit)
                "src_uri": "http://dri.freedesktop.org/libdrm/libdrm-2.4.101.tar.xz",
                                    # the provenance, that is, the place where
                                    # the upstream package came from
                "files_in_archive": 336,
                                    # The file count inside the tarball archive
                "paths": []
            },
            {
                "name": "0001-tests-modetest-automatic-configuration.patch",
                "git_sha1": null,
                "sha1_cksum": "6c84c07e589e8d188b5cbd307950e3dec5e6d396",
                "src_uri": "file://0001-tests-modetest-automatic-configuration.patch",
                                    # the provenance: in this case "unknown",
                                    # since the file was just added from a
                                    # filesystem
                "files_in_archive": false,
                                    # false, if no archive, 0 if the archive is
                                    # empty
                "paths": [
                    "_tagged/ohos/v0.1.0/openharmony-linux/stm32mp1-av96"
                ]                   # path(s) within the "files" folder where
                                    # this file is saved (the meaning of those
                                    # paths depends on the context)
            },
            {
                "name": "0002-tests-util-smtpe-increase-alpha-to-middle-band.patch",
                "git_sha1": null,
                "sha1_cksum": "09d7f85d99f23c00c0700f587b9cf8a2a8609608",
                "src_uri": "file://0002-tests-util-smtpe-increase-alpha-to-middle-band.patch",
                "files_in_archive": false,
                "paths": [
                    "_tagged/ohos/v0.1.0/openharmony-linux/stm32mp1-av96"
                ]
            },
            {
                "name": "0003-tests-modetest-set-property-in-atomic-mode.patch",
                "git_sha1": null,
                "sha1_cksum": "7c28e3a98b9201dde95708f8d28e99402a1c925a",
                "src_uri": "file://0003-tests-modetest-set-property-in-atomic-mode.patch",
                "files_in_archive": false,
                "paths": [
                    "_tagged/ohos/v0.1.0/openharmony-linux/stm32mp1-av96"
                ]
            },
            {
                "name": "0004-tests-modetest-close-crtc.patch",
                "git_sha1": null,
                "sha1_cksum": "c42fb1b703298317a79e44dc331b3e9f6cc0ac4a",
                "src_uri": "file://0004-tests-modetest-close-crtc.patch",
                "files_in_archive": false,
                "paths": [
                    "_tagged/ohos/v0.1.0/openharmony-linux/stm32mp1-av96"
                ]
            }
        ],
        "tags": [
            "ohos/v0.1.0/openharmony-linux/qemuarm/openharmony-image-extra",
            "ohos/v0.1.0/openharmony-linux/qemuarm/openharmony-image-extra-tests",
            "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra",
            "ohos/v0.1.0/openharmony-linux/qemuarm64/openharmony-image-extra-tests",
            "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-extra",
            "ohos/v0.1.0/openharmony-linux/qemux86-64/openharmony-image-extra-tests",
            "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-extra",
            "ohos/v0.1.0/openharmony-linux/qemux86/openharmony-image-extra-tests",
            "ohos/v0.1.0/openharmony-linux/stm32mp1-av96/openharmony-image-extra",
            "ohos/v0.1.0/openharmony-linux/stm32mp1-av96/openharmony-image-extra-tests"
        ]                           # tag list for this source package. The
                                    # meaning of these tags depends on the
                                    # context
    }
}

  1. Bitbake stores git repos as bare clone repos in the download directory, then copy them into the temporary build dir, checks them out against the commits indicated in SRCREV env vars, apply patches, configure and compile: so the sources found in the temporary build dir are not clean nor upstream original, and should not be used for IP compliance scanning. ↩

Copyright © Eclipse Foundation, Inc. All Rights Reserved.     Privacy Policy | Terms of Use | Copyright Agent