Skip to content
Snippets Groups Projects
Commit 973e4c6f authored by Alberto Pianon's avatar Alberto Pianon
Browse files

Merge branch 'ap/update-a4f' into 'main' 

a4f upgrade documentation

See merge request !12
parents f34fd117 fda99d32
No related branches found
No related tags found
1 merge request!12a4f upgrade documentation
Pipeline #51425 passed
Stage: compliance
    Hide whitespace changes
    Inline Side-by-side
    .reuse/dep5
    + 2
    − 0
    View file @ 973e4c6f
    ......@@ -6,6 +6,8 @@ License: CC-BY-SA-4.0 AND LicenseRef-Trademark-Notice
    Comment: authored by Peter Moser, Carlo Piana, Alberto Pianon and Martin Rabanser
    Files: deployment/fossology/upgrade/from_3.9.0_to_4.1.0/img/*
    deployment/fossology/upgrade/from_4.1.0_to_4.4.0/img/*
    postgres-upgrade-warning.png
    deployment/fossology/migration/img/*
    development/fossology/reports/img/*
    audit_workflow/img/*
    ......
    deployment/fossology/installation/install_fossology_on_debian_10.sh → deployment/fossology/installation/install_fossology_on_debian_11.sh
    + 74
    − 78
    View file @ 973e4c6f
    ......@@ -4,7 +4,7 @@
    #
    # Simple script to install Fossology from sources in Debian 10
    DEFAULT_RELEASE="4.1.0"
    DEFAULT_RELEASE="4.4.0"
    fossy_release="${FOSSY_RELEASE:-$DEFAULT_RELEASE}"
    ......@@ -12,8 +12,8 @@ cp /etc/os-release .
    chmod +x os-release
    . "./os-release"
    rm os-release
    if [[ "$NAME $VERSION" != "Debian GNU/Linux 10 (buster)" ]]; then
    echo "This script must be run only in Debian 10"
    if [[ "$NAME $VERSION" != "Debian GNU/Linux 11 (bullseye)" ]]; then
    echo "This script must be run only in Debian 11"
    exit 1
    fi
    ......@@ -31,7 +31,9 @@ echo "* INSTALLING SCRIPT DEPS... *"
    echo "***************************************************"
    apt update
    apt install -y sudo build-essential git pkg-config libpq-dev libglib2.0-dev \
    mc mawk sed software-properties-common lsb-release python3-pip
    mc mawk sed software-properties-common lsb-release php7.4-cli \
    cmake ninja-build curl cron python python3 python3-yaml python3-psycopg2 \
    python3-requests python3-pip
    apt-add-repository non-free
    apt update
    apt install -y unrar
    ......@@ -50,10 +52,16 @@ echo ""
    echo "***************************************************"
    echo "* CLONING FOSSOLOGY REPO... *"
    echo "***************************************************"
    git config --global user.email "fossy@example.com"
    git config --global user.name "Fossy"
    cd /
    git clone https://github.com/fossology/fossology.git
    cd fossology/
    git checkout tags/$fossy_release
    # https://github.com/fossology/fossology/pull/2697
    git cherry-pick 31ac4128d7986d0b07de764dcd6b6b173814080c
    # https://github.com/fossology/fossology/pull/2707
    git cherry-pick 56dedaa49ef649178b5d4d3b3ff78ee00ce3cc2d
    echo ""
    echo ""
    ......@@ -61,7 +69,6 @@ echo "***************************************************"
    echo "* CLEANING... *"
    echo "***************************************************"
    utils/fo-cleanold
    make clean
    echo ""
    echo ""
    ......@@ -76,14 +83,15 @@ echo ""
    echo "***************************************************"
    echo "* COMPILING FOSSOLOGY... *"
    echo "***************************************************"
    make
    cmake -S. -B./build -DTESTING=ON -G Ninja
    cmake --build ./build --parallel
    echo ""
    echo ""
    echo "***************************************************"
    echo "* INSTALLING FOSSOLOGY... *"
    echo "***************************************************"
    make install
    cmake --install ./build
    echo ""
    ......@@ -92,6 +100,7 @@ echo "***************************************************"
    echo "* POST INSTALL STUFF... *"
    echo "***************************************************"
    python3 -m pip install pip==21.2.2
    pip3 install --force-reinstall importlib-metadata==8.0.0
    /usr/local/lib/fossology/fo-postinstall
    a2enmod ssl
    ......@@ -103,30 +112,6 @@ awk -vRS="AllowOverride None" -vORS="AllowOverride None\n\tSSLRequireSSL" '1' \
    fossology.conf.bak | head -n -2 > fossology.conf
    service apache2 restart
    echo ""
    echo ""
    echo "***************************************************"
    echo "* removing Scancode and its deps, *"
    echo "* and installing pinned version (30.1.0) *"
    echo "***************************************************"
    su - fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 uninstall -y attrs banal binaryornot boolean charset-normalizer click colorama commoncode cryptography debian-inspector dparse extractcode extractcode-7z extractcode-libarchive fasteners fingerprints ftfy future gemfileparser html5lib idna importlib-metadata isodate jaraco.functools javaproperties Jinja2 jsonstreams license-expression lxml MarkupSafe more-itertools normality packaging parameter-expansion-patched pdfminer.six pkginfo pluggy plugincode ply publicsuffix2 pygmars Pygments pymaven-patch pyparsing rdflib requests scancode-toolkit setuptools soupsieve spdx-tools text-unidecode toml typecode typecode-libmagic urllib3 wcwidth webencodings wheel'
    su fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 install --target=$HOME/pythondeps setuptools==62.1.0 wheel==0.37.1'
    su fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 install --target=$HOME/pythondeps attrs==21.4.0 banal==1.0.6 binaryornot==0.4.4 charset-normalizer==2.0.12 click==8.1.3 colorama==0.4.4 commoncode==30.2.0 cryptography==2.6.1 debian-inspector==30.0.0 dparse==0.5.1 extractcode==30.0.0 extractcode-7z==16.5.210531 extractcode-libarchive==3.5.1.210531 fasteners==0.17.3 fingerprints==1.0.3 ftfy==6.1.1 future==0.18.2 gemfileparser==0.8.0 html5lib==1.1 idna==2.6 importlib-metadata==4.11.3 isodate==0.6.1 jaraco.functools==3.5.0 javaproperties==0.8.1 Jinja2==2.10 jsonstreams==0.6.0 license-expression==21.6.14 lxml==4.8.0 MarkupSafe==1.1.0 more-itertools==8.12.0 normality==2.3.3 packaging==21.3 parameter-expansion-patched==0.2.1b4 pdfminer.six==20220319 pkginfo==1.8.2 pluggy==0.13.1 plugincode==21.1.21 ply==3.11 publicsuffix2==2.20191221 pygmars==0.7.0 Pygments==2.12.0 pymaven-patch==0.3.0 pyparsing==3.0.8 rdflib==6.1.1 requests==2.21.0 scancode-toolkit==30.1.0 soupsieve==2.3.2.post1 spdx-tools==0.7.0a3 text-unidecode==1.3 toml==0.10.2 typecode==21.6.1 typecode-libmagic==5.39.210531 urllib3==1.24.1 wcwidth==0.2.5 webencodings==0.5.1'
    echo ""
    echo ""
    echo "***************************************************"
    echo "* PATCHING SCANCODE TO AVOID LOTS OF *"
    echo "* USELESS WARNINGS IN FOSSOLOGY LOGS *"
    echo "***************************************************"
    su - fossy -c 'echo "import warnings
    warnings.simplefilter(action=\"ignore\", category=FutureWarning)
    $(cat /home/fossy/pythondeps/scancode/cli.py)
    " > /home/fossy/pythondeps/scancode/cli.py'
    if [[ -n "$FOSSY_ENABLE_PHPPGADMIN" ]]; then
    echo ""
    ......@@ -135,7 +120,13 @@ if [[ -n "$FOSSY_ENABLE_PHPPGADMIN" ]]; then
    echo "* INSTALLING PHPPGADMIN... *"
    echo "***************************************************"
    apt install -y phppgadmin
    cd /tmp
    # phppgadmin is not available on Debian 11 anymore (why?), so we need to
    # download it from Debian 12
    wget http://ftp.de.debian.org/debian/pool/main/p/phppgadmin/phppgadmin_7.14.7+dfsg-1~bpo12+1_all.deb
    apt install -y ./phppgadmin_7.14.7+dfsg-1~bpo12+1_all.deb
    cd -
    cd /etc/apache2/conf-available/
    mv phppgadmin.conf phppgadmin.conf.bak
    sed -e 's/Require local/# Require local/' phppgadmin.conf.bak | \
    ......@@ -195,57 +186,60 @@ ln -s /usr/bin/s-nail /usr/bin/mailx
    echo ""
    echo ""
    echo "***************************************************"
    echo "* PATCHING REST API to correctly report *"
    echo "* job status *"
    echo "* FIXING ISSUE WITH SCANCODE AND MONKBULK AGENTS *"
    echo "***************************************************"
    # the bug is this one:
    # https://github.com/fossology/fossology/issues/1800#issuecomment-712919785
    # It will be solved by a complete refactoring of job rest API in this PR:
    # https://github.com/fossology/fossology/pull/1955
    # In the meantime, we need to patch it while keeping the "old" rest API logic
    sed -i -E 's/^special\[\] =/special[] = NOKILL/' /usr/local/share/fossology/scancode/scancode.conf
    cd /usr/local/share/fossology/www/ui/api/Controllers/
    patch -p1 << EOT
    --- a/JobController.php
    +++ b/JobController.php
    @@ -228,24 +228,25 @@
    \$status = "";
    \$jobqueue = [];
    + \$sql = "SELECT jq_pk from jobqueue WHERE jq_job_fk = \$1;";
    + \$statement = __METHOD__ . ".getJqpk";
    + \$rows = \$this->dbHelper->getDbManager()->getRows(\$sql, [\$job->getId()],
    + \$statement);
    /* Check if the job has no upload like Maintenance job */
    if (empty(\$job->getUploadId())) {
    - \$sql = "SELECT jq_pk, jq_end_bits from jobqueue WHERE jq_job_fk = \$1;";
    - \$statement = __METHOD__ . ".getJqpk";
    - \$rows = \$this->dbHelper->getDbManager()->getRows(\$sql, [\$job->getId()],
    - \$statement);
    if (count(\$rows) > 0) {
    - \$jobqueue[\$rows[0]['jq_pk']] = \$rows[0]['jq_end_bits'];
    - }
    - } else {
    - \$jobqueue = \$jobDao->getAllJobStatus(\$job->getUploadId(),
    - \$job->getUserId(), \$job->getGroupId());
    + \$jobqueue[] = \$rows[0]['jq_pk'];
    + }
    + } else {
    + foreach(\$rows as \$row) {
    + \$jobqueue[] = \$row['jq_pk'];
    + }
    }
    sed -i -E 's/^special\[\] =/special[] = NOKILL/' /usr/local/share/fossology/monkbulk/monkbulk.conf
    \$job->setEta(\$this->getUploadEtaInSeconds(\$job->getId(),
    \$job->getUploadId()));
    echo ""
    echo ""
    echo "***************************************************"
    echo "* IMPLEMENTING WORKAROUND TO ALLOW *"
    echo "* TO RESTART SCHEDULER VIA WebUI|API *"
    echo "***************************************************"
    - \$job->setStatus(\$this->getJobStatus(array_keys(\$jobqueue)));
    + \$job->setStatus(\$this->getJobStatus(\$jobqueue));
    }
    echo "
    #!/bin/bash
    set -e
    service fossology stop
    sleep 2
    service fossology start
    " > /usr/sbin/restart_fossology.sh
    chmod +x /usr/sbin/restart_fossology.sh
    echo "www-data ALL=(ALL) NOPASSWD: /usr/sbin/restart_fossology.sh" > /etc/sudoers.d/fossology-restart
    /**
    cd /usr/local/share/fossology/www/ui/
    patch -p1 << EOT
    --- a/admin-scheduler.php
    +++ b/admin-scheduler.php
    @@ -35,8 +35,8 @@
    "agents" => array(_("Agents"), _("Show a list of enabled agents.")),
    "verbose" => array(_("Verbose"), _("Change the verbosity level of the scheduler or a job.")),
    "stop" => array(_("Shutdown Scheduler"), _("Shutdown the scheduler gracefully and stop all background processing. This can take a while for all the agents to quit.")),
    - // "start" => array(_("Start Scheduler"), _("Start Scheduler.")),
    - // "restarts" => array(_("Restart Scheduler"), _("Restart Scheduler.")),
    + "start" => array(_("Start Scheduler"), _("Start Scheduler.")),
    + "restarts" => array(_("Restart Scheduler"), _("Restart Scheduler.")),
    "restart" => array(_("Unpause a job"), _("Unpause a job.")),
    "pause" => array(_("Pause a running job"), _("Pause a running job.")),
    "priority" => array(_("Priority"), _("Change the priority of a job."))
    @@ -197,9 +197,9 @@
    function StartScheduler(\$operation = '')
    {
    if (\$operation) {
    - \$command = "/etc/init.d/fossology restart >/dev/null 2>&1";
    + \$command = "sudo /usr/sbin/restart_fossology.sh >/dev/null 2>&1";
    } else {
    - \$command = "/etc/init.d/fossology start >/dev/null 2>&1";
    + \$command = "sudo /usr/sbin/restart_fossology.sh >/dev/null 2>&1";
    }
    \$lastline = system(\$command, \$rc);
    if (\$rc) {
    EOT
    cd -
    echo ""
    ......@@ -254,8 +248,10 @@ echo "***************************************************"
    echo "* ENABLING FOSSOLOGY SCHEDULER... *"
    echo "***************************************************"
    systemctl enable fossology.service
    systemctl start fossology.service
    systemctl daemon-reload
    systemctl enable --now fossology
    service fossology restart
    su - www-data -s /bin/bash -c 'sudo /usr/sbin/restart_fossology.sh'
    echo ""
    echo ""
    ......
    deployment/fossology/upgrade/from_4.1.0_to_4.4.0/README.md
    + 128
    − 25
    View file @ 973e4c6f
    ......@@ -2,7 +2,7 @@
    title: "Upgrade from Fossology 4.1.0 to 4.4.0: activity log and report"
    author: Alberto Pianon <pianon@array.eu>
    date: "2024-02-12"
    lastmod: "2024-02-24"
    lastmod: "2024-05-06"
    SPDX-License-Identifier: CC-BY-SA-4.0 AND LicenseRef-Trademark-Notice
    SPDX-FileCopyrightText: Alberto Pianon <pianon@array.eu>
    ---
    ......@@ -17,7 +17,7 @@ SPDX-FileCopyrightText: Alberto Pianon <pianon@array.eu>
    - [x] Check if system configuration tweaks are still applied
    - [x] verify and check if everything works correctly
    - [ ] Upgrade fossology-python package in aliens4friends (include it in docker image) and update and test api calls in a4f code
    - [ ] implement missing customizations (if any): eg. job status API
    - [ ] implement missing customizations (if any): eg. job status API
    ## 1. Check if installation/Upgrade instructions in the official wiki changed since the last time
    ......@@ -55,7 +55,7 @@ None relevant, apparently; only license IDs fixes for SPDX compliance (see [dbmi
    | ------ | ------ |
    | [Adjusting the Kernel](https://github.com/fossology/fossology/wiki/Configuration-and-Tuning#adjusting-the-kernel) | yes |
    | [Tweaking postgresql](https://github.com/fossology/fossology/wiki/Configuration-and-Tuning#preparing-postgresql) | yes |
    | [PHP configuration](https://github.com/fossology/fossology/wiki/Configuration-and-Tuning#configuring-php) (see [here](../../installation/install_fossology_on_debian_10.sh) line 155) | yes |
    | [PHP configuration](https://github.com/fossology/fossology/wiki/Configuration-and-Tuning#configuring-php) (see [here](../../installation/install_fossology_on_debian_10.sh) line 155) | yes |
    | mail configuration (see [here](../../installation/install_fossology_on_debian_10.sh) line 167) | yes |
    # 3. create a backup of the current instance (only database and configuration files, but including php and postgres config files)
    ......@@ -65,14 +65,14 @@ PHP_PATH=$(php --ini | awk '/\/etc\/php.*\/cli$/{print $5}')
    phpIni="$(realpath ${PHP_PATH}/../apache2/php.ini)"
    tar czvf 20240212_fossology_code_and_config_backup.tgz \
    /var/local/lib/fossology \
    /var/local/cache/fossology \
    /usr/local/etc/fossology \
    /usr/local/lib/fossology \
    /usr/local/share/fossology \
    $phpIni \
    /etc/postgresql/11/main \
    /var/lib/postgresql/11/main/postgresql.auto.conf
    /var/local/lib/fossology \
    /var/local/cache/fossology \
    /usr/local/etc/fossology \
    /usr/local/lib/fossology \
    /usr/local/share/fossology \
    $phpIni \
    /etc/postgresql/11/main \
    /var/lib/postgresql/11/main/postgresql.auto.conf
    su - postgres -c 'pg_dumpall' | gzip > 20240212_fo_dbbackup.gz
    # create also an internal copy
    ......@@ -124,15 +124,15 @@ log (excerpt):
    ```
    [...]
    The following NEW packages will be installed:
    cmake cmake-data libcppunit-1.14-0 libcppunit-dev libomp-dev librhash0 libuv1 libzstd-dev ninja-build
    cmake cmake-data libcppunit-1.14-0 libcppunit-dev libomp-dev librhash0 libuv1 libzstd-dev ninja-build
    0 upgraded, 9 newly installed, 0 to remove and 1 not upgraded.
    [...]
    The following NEW packages will be installed:
    php7.3-sqlite3
    php7.3-sqlite3
    0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
    [...]
    The following NEW packages will be installed:
    zstd
    zstd
    0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
    [...]
    ```
    ......@@ -146,7 +146,7 @@ cmake --install ./build
    the last command returns the following warning:
    ```
    Needs CMake version greater than 3.19.0 for packaging FOSSology. Skipping
    Needs CMake version greater than 3.19.0 for packaging FOSSology. Skipping
    ```
    and when trying to do
    ......@@ -165,7 +165,7 @@ None of the scripts supposed to be installed into `/usr/local/lib/fossology/` ar
    # 5. Debian upgrade
    Let's upgrade debian from version 10 to version 11, because Debian 11 is the version used in Fossology's current docker imag
    Let's upgrade debian from version 10 to version 11, because Debian 11 is the version used in Fossology's current docker image
    ## 5.1. System upgrade
    ......@@ -308,11 +308,11 @@ Php configuration needs to be applied again, instead (there's a new php version)
    PHP_PATH=$(php --ini | awk '/\/etc\/php.*\/cli$/{print $5}')
    phpIni="${PHP_PATH}/../apache2/php.ini"
    sed \
    -i.bak \
    -e "s/upload_max_filesize = 700M/upload_max_filesize = 1000M/" \
    -e "s/post_max_size = 701M/post_max_size = 1004M/" \
    -e "s/memory_limit = 702M/memory_limit = 3030M/" \
    $phpIni
    -i.bak \
    -e "s/upload_max_filesize = 700M/upload_max_filesize = 1000M/" \
    -e "s/post_max_size = 701M/post_max_size = 1004M/" \
    -e "s/memory_limit = 702M/memory_limit = 3030M/" \
    $phpIni
    ```
    Fix "Sudo: unable to resolve host" error, again:
    ......@@ -347,14 +347,14 @@ Apparently it works, but when opening any upload it displays a blank page. After
    > Warning: pg_prepare(): Query failed: ERROR: column license_ref.rf_spdx_id does not exist LINE 2: FROM ( SELECT license_ref.rf_shortname, license_ref... ^ in /usr/local/share/fossology/lib/php/Db/Driver/Postgres.php on line 49
    >
    >
    > Fatal error: Uncaught Fossology\Lib\Exception: error executing: SELECT rf_shortname AS license_shortname, rf_spdx_id AS spdx_id, rf_pk, count(*) AS count, count(distinct pfile_ref.pfile_fk) as "unique" FROM ( SELECT license_ref.rf_shortname, license_ref.rf_spdx_id, license_ref.rf_pk, license_file.fl_pk, license_file.agent_fk, license_file.pfile_fk FROM license_file JOIN license_ref ON license_file.rf_fk = license_ref.rf_pk) AS pfile_ref RIGHT JOIN uploadtree_a UT ON pfile_ref.pfile_fk = UT.pfile_fk AND agent_fk=ANY($4) WHERE (rf_shortname IS NULL OR rf_shortname NOT IN ('Void')) AND upload_fk=$1 AND (UT.lft BETWEEN $2 AND $3) AND UT.ufile_mode&(3<<\28)=0 GROUP BY license_shortname, spdx_id, rf_pk -- Fossology\Lib\Dao\LicenseDao::getLicenseHistogram.uploadtree_a.36,33,38 ERROR: column license_ref.rf_spdx_id does not exist LINE 2: FROM ( SELECT license_ref.rf_shortname, license_ref... ^ in /u in /usr/local/share/fossology/lib/php/Db/DbManager.php on line 133
    Tried to re-run `/usr/local/lib/fossology/fo-postinstall` and it apparently solved the problem (db migration has been successfully performed).
    The issue was due to the fact that in the existing installation the db password, stored in Db.conf, was not the default one ('fossy'). When installing/upgrading Fossology, Db.conf was rewritten with the default password. When running fo-postinstall an "unable to connect to database" error occurred, so I restored the original password in Db.conf and restarted Fossology: it apparently worked, but I didn't realize that I had to re-run fo-postinstall to perform the db migration which was stopped because of the db login error.
    The issue was due to the fact that in the existing installation the db password, stored in Db.conf, was not the default one ('fossy'). When installing/upgrading Fossology, Db.conf was rewritten with the default password. When running fo-postinstall an "unable to connect to database" error occurred, so I restored the original password in Db.conf and restarted Fossology: it apparently worked, but I didn't realize that I had to re-run fo-postinstall to perform the db migration which was stopped because of the db login error.
    Opened an issue upstream: https://github.com/fossology/fossology/issues/2683
    Opened an issue upstream: <https://github.com/fossology/fossology/issues/2683>
    ## 8.1. Upload zlib package and run all agents
    ......@@ -393,12 +393,115 @@ change the last line to:
    special[] = NOKILL
    ```
    # 11. Backport of upstream fixes:
    ## 11. Workaround to activate the scheduler restart functionality in REST API
    Beacuse of [this issue](https://github.com/fossology/fossology/issues/2684) it may be required to restart the scheduler via REST API when it hangs (or before or after every upload, as a preventive measure).
    However the scheduler restart functionality is commented out Fossology's code, probably because it was not working properly for permissions issues.
    To activate it, first we need to create a restart script at `/usr/sbin/restart_fossology.sh` (all the following commands need to be run as root)):
    ```bash
    #!/bin/bash
    set -e
    service fossology stop
    sleep 2
    service fossology start
    ```
    (we do not use `service fossology restart` because it may not work properly; we stop the service, wait a bit, then start it again).
    Then we need to change the permissions of the script:
    ```bash
    chmod +x /usr/sbin/restart_fossology.sh
    ```
    Then we need to allow the www-data user to run the script with sudo without password:
    ```bash
    echo "www-data ALL=(ALL) NOPASSWD: /usr/sbin/restart_fossology.sh" > /etc/sudoers.d/fossology-restart
    ```
    Now we check if we are able to run it:
    ```bash
    su - www-data -s /bin/bash -c 'sudo /usr/sbin/restart_fossology.sh'
    ```
    If it works, we can proceed.
    We need to edit `/usr/local/share/fossology/www/ui/admin-scheduler.php`. First, we need to uncomment lines 38-39:
    ```php
    "start" => array(_("Start Scheduler"), _("Start Scheduler.")),
    "restarts" => array(_("Restart Scheduler"), _("Restart Scheduler.")),
    ```
    then we need to change the start/restart commands in lines 200-202:
    ```php
    if ($operation) {
    $command = "sudo /usr/sbin/restart_fossology.sh >/dev/null 2>&1";
    } else {
    $command = "sudo /usr/sbin/restart_fossology.sh >/dev/null 2>&1";
    }
    ```
    We keep the same command for both start and restart, because the `service fossology start` command may not work properly.
    # 12. Backport of upstream fixes:
    - We raised [this issue](https://github.com/fossology/fossology/issues/2693) that made the file info endpoint fail in REST API, and we got accepted our fix in this [PR](https://github.com/fossology/fossology/pull/2697). The corresponding patch has been backported to our Fossology instance
    - We raised [this issue](https://github.com/fossology/fossology/issues/2715) that made spdx report generation occasionally fail, but that had been already been fixed by this [PR](https://github.com/fossology/fossology/pull/2707). The corresponding patch has been backported to our Fossology instance
    # 13. Upgrade fossology-python package in aliens4friends
    ## 13.1. Check if the API has changed (fossology-python 1.5.0 -> 3.2.1)
    | method or object | changed in latest version |
    | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
    | `fossology.list_jobs(upload=upload, page_size=2000)` | apparently the same |
    | `fossology.upload_file()` | almost the same, only access_level *optional* input param now needs to be an object and not a string |
    | `Job.queueDate`, `Job.status` | apparently the same |
    | `fossology.rootFolder` | apparently the same |
    | `fossology.create_folder(parent_obj, name)` | apparently the same |
    | `fossology.list_uploads()` | apparently the same |
    | `Upload` | almost the same, removed filesize and filesha1 (only hash), added assignee-related properties |
    | `fossology.detail_folder(upload.folderid)` (returns folder obj) | same |
    | `Folder` | same |
    | `fossology.schedule_jobs(folder, upload, specs, wait=True)` | same |
    | `fossology.upload_summary(upload)` | same |
    | `Summary` | same |
    | `fossology.session.get` (used to get licenses and raw json summary, not using python wrapper method) | same, also json output is the same |
    | `fossology.api` | same |
    | `fossology.generate_report(upload=upload,report_format=ReportFormat.SPDX2TV)` (returns rep_id) | same |
    | `fossology.download_report(rep_id)` | same |
    It seems that nothing has changed in the API, so we can safely upgrade the package in aliens4friends and test it.
    ACTUALLY Summary json response has changed, so we need to update the code to handle the new format. TODO: file an issue upstream
    The only problem is the python version supported by fossology-python which is 3.10, while in aliens4friends it's 3.8 (also for compatibility with scancode v3.2.3)
    Scancode 3.2.3 cannot run on 3.10, but a special virtualenv can be created for scancode with python 3.8. This way, we can use the latest fossology-python version in aliens4friends.
    ## 13.2 Used WebUI Functionalities not exposed in API
    | functionality | added to API in 4.4.0 | added to fossology-python 3.2.1 |
    | ------------- | --------------------- | ------------------------------- |
    | reportImport | yes | no |
    | rename upload | no :( | no |
    As a temporary solution, we may keep using the WebUI for these functionalities, and switch to REST API when both will be fully implemented both in the API and in fossology-python.
    ## 13.3 Preliminary conclusions
    The upgrade of fossology-python package in aliens4friends should not require modifications to the code, but some possible hidden differences in the API behavior and possible issues due to the upgrade to python 3.10 could be discovered only by testing it.
    0% or .
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment

    Copyright © Eclipse Foundation, Inc. All Rights Reserved.     Privacy Policy | Terms of Use | Copyright Agent