|
|
|
|
|
# ORT Setup and execution
|
|
|
This page describes how to execute analyses on projects or specific repositories on the current infrastructure. We describe the remote process (via Jenkins) and the local execution on the VM (via a shell).
|
|
|
|
|
|
This is the summary of the current setup of ORT and required tools on the Eclipse VM at `ort-vm1.eclipse.org`.
|
|
|
## Jenkins remote scan
|
|
|
|
|
|
The Jenkins instance for ORT is located at https://foundation.eclipse.org/ci/infra/job/ort/. It is accessible to all read-only, only authorized people can actually start an analysis.
|
|
|
|
|
|
## VM setup
|
|
|
Note that the Jenkins instance offers two folders for ORT scans jobs: ORT Prod and ORT Staging.
|
|
|
* ORT Prod is the officially supported branch; it is configured to use the `main` branch of the Git repository and runs on the `ort-vm1` VM.
|
|
|
* ORT Staging is a test branch used to validate new developments, may fail or provide flawed results, and is not officially supported. It uses the `staging` Git branch and runs jobs on the `ortstaging-vm1` VM.
|
|
|
|
|
|
There are two entry points to start an analysis on [Jenkins Prod](https://foundation.eclipse.org/ci/infra/job/ort/job/ORT_Prod/):
|
|
|
* [run_ort](https://foundation.eclipse.org/ci/infra/job/ort/job/ORT_Prod/job/run_ort/) analyses an existing Eclipse project, including all its repositories. It takes as parameter the project ID of the project, like `ecd.che` or `polarys.capella`. When the full project is analysed results are published on https://ort-vm1.eclipse.org.
|
|
|
* [run_ort_repo](https://foundation.eclipse.org/ci/infra/job/ort/job/ORT_Prod/job/run_ort_repo/) can analyse any Git repository. It takes as parameter a full git repository URL, like `https://github.com/eclipse-opensmartclide/smartclide.git`. Any publicly accessible URL should work. When the repo has been analysed results are published on https://ort-vm1.eclipse.org with a timestamp of the run as project.
|
|
|
|
|
|
## VM local scan
|
|
|
|
|
|
Main directories:
|
|
|
* All **applications** (scancode, ort) are in `/ort/`: `/ort/scancode-toolkit-30.1.0/`, `/ort/ort/`.
|
|
|
* All **applications** (ort) are in `/ort/`.
|
|
|
* **Configuration** is cloned in `/ort/eclipsefdn-ort/`.
|
|
|
* **Project code** is cloned in `/data/projects/`.
|
|
|
* **Results** are stored in `/data/results/`.
|
|
|
|
|
|
Notes:
|
|
|
* We install the software as `root`, but chown and execute it as user `ort`.
|
|
|
* All operations (configuration, runs..) are now meant to be executed as `ort`.
|
|
|
|
|
|
|
|
|
## Start a run
|
|
|
|
|
|
Runs can last for a long time (e.g. several hours). It is highly recommended to execute runs with `nohup` or `screen`.
|
|
|
|
|
|
### Connect to the VM host
|
|
|
|
|
|
SSH into `ort-vm1.eclipse.org`.
|
|
|
|
|
|
If you don't have an account, please [submit an issue](https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort/-/issues) and provide us with your public ssh key.
|
|
|
|
|
|
### Clone your project
|
|
|
|
|
|
If the project ou want to run is not already cloned, then clone it:
|
|
|
* You will need to connect as ort or genie.infra to actually start a local scan.
|
|
|
* Runs can last for a long time (e.g. several hours). It is highly recommended to execute runs with `nohup` or `screen`.
|
|
|
|
|
|
Steps to execute a run:
|
|
|
* Clone or pull the repository to be analysed.
|
|
|
* Execute the script `/ort/eclipsefdn-ort/scripts/run_ort.sh`. Its usage is as follows: \
|
|
|
```
|
|
|
ort@ort-vm1:~$ cd /data/projects/
|
|
|
ort@ort-vm1:/data/projects$ ls
|
|
|
acumos-nexus-client jgit kie-wb-distributions org.eclipse.epsilon org.eclipse.tracecompass smartclide-broker smartclide-context tycho
|
|
|
che kie-wb-common org.eclipse.emf.ecp.core org.eclipse.sirius smartclide-api-gateway smartclide-cicd theia
|
|
|
ort@ort-vm1:/data/projects$ git clone https://git.eclipse.org/r/cdt/org.eclipse.cdt
|
|
|
Cloning into 'org.eclipse.cdt'...
|
|
|
|
|
|
[SNIP]
|
|
|
Usage: /ort/eclipsefdn-ort/scripts/run_ort.sh <dir_project> <repo> <dir_results>
|
|
|
```
|
|
|
* Publish the results using the script `/ort/eclipsefdn-ort/scripts/run_extract.sh` without arguments. \
|
|
|
|
|
|
If it's already there, simply update (pull) it.
|
|
|
|
|
|
### Execute project run
|
|
|
|
|
|
There is a single script to run the pre-configured analysis, located in `/ort/eclipsefdn-ort/scripts/`. its usage is as follows:
|
|
|
|
|
|
An example run would look like the following:
|
|
|
```
|
|
|
Usage: /ort/eclipsefdn-ort/scripts/run_ort.sh <dir_project> <dir_results>
|
|
|
```
|
|
|
|
|
|
To analyse a project simply execute it with the path to the project and results directory:
|
|
|
genie.infra@ort-vm1:/ort/eclipsefdn-ort$ /ort/eclipsefdn-ort/scripts/run_ort.sh /data/projects/eclipse.pde/ eclipse.pde /data/results/
|
|
|
Start scan at 2023-01-24 06:43
|
|
|
- Using docker ort image [ort].
|
|
|
- Log written to [/data/results/eclipse.pde/eclipse.pde/eclipse.pde_eclipse.pde_202301240643.log].
|
|
|
- Scripts dir [/ort/eclipsefdn-ort/scripts].
|
|
|
- Working on project [/data/projects/eclipse.pde//eclipse.pde].
|
|
|
|
|
|
```
|
|
|
ort@ort-vm1:~$ time /ort/eclipsefdn-ort/scripts/run_ort.sh /data/projects/org.eclipse.cdt/ /data/results/
|
|
|
Start scan at 2022-03-07 09:16
|
|
|
- Using ort from [ort].
|
|
|
- Log written to [/data/results/org.eclipse.cdt_202203070916.log].
|
|
|
- Working on project [/data/projects/org.eclipse.cdt/].
|
|
|
* Checking ort.yml file at [/ort/eclipsefdn-ort/conf/repositories/eclipse.pde/eclipse.pde/].
|
|
|
- Found. Using it for next steps.
|
|
|
|
|
|
* Start analyzer at 2022-03-07 09:16
|
|
|
* Start analyzer at 2023-01-24 06:43
|
|
|
docker run -v /data/projects/eclipse.pde//eclipse.pde:/project -v /data/cache/ort/:/ort_cache/ -v /ort/eclipsefdn-ort/conf:/home/ort/.ort/config/ -v /data/results/eclipse.pde/eclipse.pde/202301240643:/output --rm -e JAVA_OPTS=-Xmx10g ort --info analyze -f JSON --package-curations-dir /home/ort/.ort/config//curations/ --clearly-defined-curations --ort-curations --repository-configuration-file /home/ort/.ort/config//repositories/eclipse.pde/eclipse.pde/ort.yml -i /project -o /output
|
|
|
|
|
|
[SNIP]
|
|
|
|
|
|
* Finish reporter at 2023-01-24 06:47
|
|
|
|
|
|
Reading JSON file from /data/results/eclipse.pde/eclipse.pde/202301240643/evaluation-result.json.
|
|
|
- Working on 'Maven:org.eclipse.pde:eclipse.pde:4.27.0-SNAPSHOT' '{'rule': 'PROJECT_LICENSE_CHECK', 'pkg': 'Maven:org.eclipse.pde:eclipse.pde:4.27.0-SNAPSHOT', 'license': 'MIT-open-group', 'license_source': 'DETECTED', 'severity': 'HINT', 'message': "License MIT-open-group of project 'Maven:org.eclipse.pde:eclipse.pde:4.27.0-SNAPSHOT' is unclassified.", 'how_to_fix': 'Classify MIT-open-group as either approved or restricted.'}' 'Maven:org.eclipse.pde:eclipse.pde:4.27.0-SNAPSHOT'.. No declared license found on clearlydefined.
|
|
|
- Working on 'Maven:org.eclipse.pde:eclipse.pde.build:4.27.0-SNAPSHOT' '{'rule': 'PROJECT_LICENSE_CHECK', 'pkg': 'Maven:org.eclipse.pde:eclipse.pde.build:4.27.0-SNAPSHOT', 'license': 'MIT-open-group', 'license_source': 'DETECTED', 'severity': 'HINT', 'message': "License MIT-open-group of project 'Maven:org.eclipse.pde:eclipse.pde.build:4.27.0-SNAPSHOT' is unclassified.", 'how_to_fix': 'Classify MIT-open-group as either approved or restricted.'}' 'Maven:org.eclipse.pde:eclipse.pde.build:4.27.0-SNAPSHOT'.. No declared license found on clearlydefined.
|
|
|
- Working on 'Maven:org.eclipse.pde:eclipse.pde.ui:4.27.0-SNAPSHOT' '{'rule': 'PROJECT_LICENSE_CHECK', 'pkg': 'Maven:org.eclipse.pde:eclipse.pde.ui:4.27.0-SNAPSHOT', 'license': 'NOASSERTION', 'license_source': 'DETECTED', 'severity': 'HINT', 'message': "License NOASSERTION of project 'Maven:org.eclipse.pde:eclipse.pde.ui:4.27.0-SNAPSHOT' is unclassified.", 'how_to_fix': 'Classify NOASSERTION as either approved or restricted.'}' 'Maven:org.eclipse.pde:eclipse.pde.ui:4.27.0-SNAPSHOT'.. No declared license found on clearlydefined.
|
|
|
- Working on 'Maven:org.eclipse.pde:org.eclipse.pde:3.13.2200-SNAPSHOT' '{'rule': 'PROJECT_LICENSE_CHECK', 'pkg': 'Maven:org.eclipse.pde:org.eclipse.pde:3.13.2200-SNAPSHOT', 'license': 'NOASSERTION', 'license_source': 'DETECTED', 'severity': 'HINT', 'message': "License NOASSERTION of project 'Maven:org.eclipse.pde:org.eclipse.pde:3.13.2200-SNAPSHOT' is unclassified.", 'how_to_fix': 'Classify NOASSERTION as either approved or restricted.'}' 'Maven:org.eclipse.pde:org.eclipse.pde:3.13.2200-SNAPSHOT'.. No declared license found on clearlydefined.
|
|
|
|
|
|
Found 0 curations.
|
|
|
Read curations from /ort/eclipsefdn-ort/conf/curations/clearlydefined_curations.yml.
|
|
|
Write curations to /ort/eclipsefdn-ort/conf/curations/clearlydefined_curations.yml.
|
|
|
genie.infra@ort-vm1:/ort/eclipsefdn-ort$
|
|
|
```
|
|
|
|
|
|
The scripts logs all output to `${DIR_RESULTS}/<project>_<date>.log`, and the analysis result files are stored in `${DIR_RESULTS}/<project>_<date>/`.
|
|
|
The script logs all output to `${DIR_RESULTS}/<project>/<repo>/<project>_<repo>_<datetime>.log`, and the analysis result files are stored in `${DIR_RESULTS}/<project>/<repo>/<datetime>/`.
|
|
|
|
|
|
## Analysing several projects at once
|
|
|
### Analysing several projects at once
|
|
|
|
|
|
Following command comes in handy:
|
|
|
|
... | ... | |