Commit 0c82f873 authored by Boris Baldassari's avatar Boris Baldassari
Browse files

Merge branch 'dev' into 'main'

Merge dev into main

See merge request !3
parents c1ed7ae4 e9299298
......@@ -3,6 +3,17 @@
This repository holds information about Eclipse's ORT setup and deployment.
More information can be found at:
* The [project repository itself](https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort)
- We use the [issues there](https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort/-/issues) to track requirements, progress and bugs.
- The `main` branch is the stable, deployed version whereas `dev` is the on-going development branch.
- Documentation is [available in the wiki](https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort/-/wikis/home).
* The static web site found on the VM's standard port: \
https://ort-vm1.eclipse.org
For any question or inquiry, please submit an issue or get in touch with Boris.
## Structure of the repository
......@@ -12,20 +23,23 @@ $ tree
├── conf # All configuration files needed
│   ├── ... # to execute ORT with our reqs
│   └── ort.conf
├── docs # Documentation for the setup
│   └── ort_install_notes.md
├── README.md
└── scripts # Scripts to automate runs and
├── ... # conf as much as possible
└── run_ort.sh
```
The repository is self-container, everything required to run ORT with our
The repository is self-contained, everything required to run ORT with our
own configuration and setup should be included.
## Setup
Edit the environment variables in the script `run_ort.sh` to point to the
We provide 3 scripts:
* `setup_ort_conf.sh` creates the proper configuration files for licenses classification and curations from IPZilla. Only needed when there is a change in the configuration files or for a new setup.
* `run_ort.sh` runs the ORT analysis on a specific project.
* `run_extract.sh` parses the log files of all runs and publishes a static website with all projects, runs and errors. It then copies the published website into `/var/www/html/` for direct access through the host's web serving Apache.
Check and edit accordingly the environment variables in the script `run_ort.sh` to point to the
actual location of the repository and binaries (ORT, ScanCode) and execute
the script:
......@@ -36,3 +50,5 @@ $ time bash run_ort.sh /path/to/project/ /path/to/results/
Execution time varies from a few minutes to a few hours, depending on the
size of the project.
**Learn more about execution in the dedicated document: [Executing ORT](Executing ORT).**
- id: "NPM::argparse:1.0.10"
curations:
comment: "Manually checked issue, it is a false-positive. License available at https://www.npmjs.com/package/argparse/v/1.0.10"
concluded_license: "MIT"
- id: "NPM::chokidar:3.5.2"
curations:
comment: "Manually checked issue, it is a false-positive. License available at https://www.npmjs.com/package/chokidar/v/3.5.2"
concluded_license: "MIT"
- id: "NPM::diff:3.5.0"
curations:
comment: "Manually checked issue, it is a false-positive. License available at https://www.npmjs.com/package/diff/v/3.5.0"
concluded_license: "BSD-3-Clause"
# - id: "NPM::is-extglob:2.1.1"
# curations:
# comment: "License available at https://www.npmjs.com/package/is-extglob/v/2.1.1"
# concluded_license: "MIT"
# - id: "NPM::jszip:3.7.1"
# curations:
# comment: "License available at https://www.npmjs.com/package/jszip/v/3.7.1"
# concluded_license: "MIT OR GPL-3.0"
# - id: "NPM::selenium-webdriver:3.6.0"
# curations:
# comment: "License available at https://www.npmjs.com/package/selenium-webdriver/v/3.6.0"
# concluded_license: "Apache-2.0"
This diff is collapsed.
- id: "NPM:@eclipse-che:che-e2e:7.45.0-SNAPSHOT"
curations:
comment: "Wrong text parsing."
declared_license_mapping:
"LicenseRef-scancode-unknown-license-reference": "EPL-2.0"
- id: "Maven:org.eclipse.cdt:org.eclipse.cdt-parent:10.6.0-SNAPSHOT"
curations:
comment: "Wrong text parsing."
declared_license_mapping:
"EPL-2.0</a>": "EPL-2.0"
File mode changed from 100644 to 100755
File mode changed from 100644 to 100755
......@@ -2,6 +2,7 @@
"ignored": {
"LicenseRef-scancode-dco-1.1": "Not a license, wrong identification by scancode.",
"LicenseRef-scancode-generic-cla": "Not a license, wrong identification by scancode.",
"LicenseRef-scancode-warranty-disclaimer": "Not a license, wrong identification by scancode."
"LicenseRef-scancode-warranty-disclaimer": "Not a license, wrong identification by scancode.",
"LicenseRef-scancode-unknown-license-reference": "False-positive."
}
}
File mode changed from 100644 to 100755
File mode changed from 100644 to 100755
File mode changed from 100644 to 100755
......@@ -12,7 +12,6 @@ categories:
that the source code of the packages needs to be provided.
name: include-source-code-offer-in-notice-file
- name: ef-approved
- name: ef-approved-priv
- name: ef-ignored
categorizations:
- categories:
......@@ -64,7 +63,7 @@ categorizations:
- ef-approved
id: BSD-2-Clause
- categories:
- ef-approved-priv
- ef-approved
id: BSD-2-Clause-FreeBSD
- categories:
- permissive
......@@ -110,10 +109,10 @@ categorizations:
- ef-approved
id: CDDL-1.1
- categories:
- ef-approved-priv
- ef-approved
id: EPL-1.0
- categories:
- ef-approved-priv
- ef-approved
id: EPL-2.0
- categories:
- copyleft-limited
......@@ -229,7 +228,7 @@ categorizations:
- ef-approved
id: MIT
- categories:
- ef-approved-priv
- ef-approved
id: MIT-0
- categories:
- permissive
......@@ -291,7 +290,7 @@ categorizations:
- ef-approved
id: W3C
- categories:
- ef-approved-priv
- ef-approved
id: WTFPL
- categories:
- ef-approved
......@@ -327,10 +326,10 @@ categorizations:
- ef-approved
id: CC-By-2.5
- categories:
- ef-approved-priv
- ef-approved
id: CC-By-3.0
- categories:
- ef-approved-priv
- ef-approved
id: CC-By-4.0
- categories:
- ef-approved
......@@ -384,13 +383,13 @@ categorizations:
- ef-approved
id: ZPL-2.1
- categories:
- ef-approved-priv
- ef-approved
id: UNLICENSE
- categories:
- ef-approved-priv
- ef-approved
id: ARTISTIC-2.0
- categories:
- ef-approved-priv
- ef-approved
id: OBSD
- categories:
- ef-ignored
......
ort {
analyzer {
allowDynamicVersions = true
}
scanner {
skipConcluded = true
options {
# A map of maps from scanner class names to scanner-specific key-value pairs.
# At the example of applying custom options for ScanCode, this would look like:
ScanCode {
commandLine = --copyright --license --info --strip-root -n 4 --timeout 300 --license-score 100
parseLicenseExpressions = true
# Criteria for matching stored scan results. These can be configured for any scanner that uses semantic
# versioning. Note that the "maxVersion" is exclusive and not part of the range of accepted versions.
minVersion = "3.2.1-rc2"
maxVersion = "30.2.0"
}
}
storages {
fileBasedStorage {
backend {
......@@ -9,18 +32,23 @@ ort {
}
}
}
clearlyDefined {
serverUrl = "https://api.clearlydefined.io"
}
}
storageReaders: [
"fileBasedStorage",
"clearlyDefined"
ignorePatterns = [
"**/META-INF/DEPENDENCIES",
"**/META-INF/DEPENDENCIES",
"**/META-INF/NOTICE",
"**/META-INF/NOTICE.txt",
"**/*.svg",
"package-lock.json"
]
storageReaders = [
"fileBasedStorage"
]
storageWriters: [
storageWriters = [
"fileBasedStorage"
]
}
}
\ No newline at end of file
}
rule_violations:
- message: ".*LicenseRef-scancode-public-domain.*NPM::jszip:3.7.1.*"
reason: "EXAMPLE_OF_EXCEPTION"
comment: "See CQ-XXXXXX"
# ORT installation guide
This is the summary of the installation of ORT and required tools on host `ort-vm1.eclipse.org`.
Notes:
* We install the software as root, but chown and execute it as user ort.
* All applications (scancode, ort) are in `/ort/`.
* Configuration is cloned in `/ort/eclipsefdn-ort/`.
* Project code is cloned in `/data/projects/`.
* Results are stored in `/data/results/`.
## System requirements
As root:
```
apt-get install prometheus-node-exporter htop
apt-get install openjdk-17-jdk nodejs git python3-virtualenv python3-pip npm subversion cvs mercurial
npm install --global yarn
```
## Install scancode
As root:
```
cd /ort
wget https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py39-linux.tar.xz
tar xJf scancode-toolkit-30.1.0_py39-linux.tar.xz
chown -R ort:users scancode-toolkit-30.1.0
```
As user ort:
```
cd /ort/scancode-toolkit-30.1.0
./scancode --help
```
## Install Conan
As root:
```
pip install conan
```
## Install ORT
As root:
```
cd /ort
git clone --recurse-submodules https://github.com/oss-review-toolkit/ort.git
chown -R ort:users ort
```
As user ort:
```
cd /ort/ort
export JAVA_HOME=/lib/jvm/java-17-openjdk-amd64/
./gradlew installDist
./gradlew cli:run --args="requirements"
```
Add binaries to PATH:
```
cd /usr/local/bin/
ln -s /ort/scancode-toolkit-30.1.0/scancode .
ln -s /ort/ort/cli/build/install/ort/bin/ort .
```
Check that they can be found:
```
ort@ort-vm1:/usr/local/bin$ ort requirements
________ _____________________
\_____ \\______ \__ ___/ the OSS Review Toolkit, version a056b54c72.
/ | \| _/ | |
/ | \ | \ | | Running 'requirements' under Java 17.0.2 on Linux with
\_______ /____|_ / |____| 8 CPUs and a maximum of 4004 MiB of memory.
\/ \/
Environment variables:
ORT_CONFIG_DIR = /home/ort/.ort/config
ORT_DATA_DIR = /home/ort/.ort
SHELL = /bin/bash
TERM = screen.xterm-256color
JAVA_HOME = /lib/jvm/java-17-openjdk-amd64/
Scanners:
- Askalono: Requires 'askalono' in no specific version. Tool not found.
- BoyterLc: Requires 'lc' in no specific version. Tool not found.
- Licensee: Requires 'licensee' in no specific version. Tool not found.
* ScanCode: Requires 'scancode' in no specific version. Found version 30.1.0.
[SNIP]
```
## Prepare for run
Clone the Eclipse ORT git repository:
```
git clone https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort.git
chown -R ort:users eclipsefdn-ort/
```
Add following lines to ~ort/.bashrc
```
export EDITOR=emacs
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export ORT_CONFIG_DIR=/ort/eclipsefdn-ort/conf
```
Increase allocated Java Memory by editing `ORT_HOME/cli/build/install/ort/bin/ort` and set DEFAULT_JVM_OPTS:
```
DEFAULT_JVM_OPTS="-Xmx12192M"
```
## Execute first run
Clone a batch of git repositories:
```
cd /data/projects/
git clone https://git.eclipse.org/r/jgit/jgit
git clone https://github.com/eclipse-theia/theia
git clone https://github.com/eclipse/tycho
git clone https://github.com/eclipse/che.git
```
Execute first run to check everything is in place:
```
ort@ort-vm1:/data/projects$ time bash /ort/eclipsefdn-ort/scripts/run_ort.sh /data/projects/che/ /data/results/
Start scan at 2022-02-24 10:40
- Using ort from [ort].
- Log written to [/data/results/che_202202241040.log].
- Working on project [/data/projects/che/].
* Start analyzer at 2022-02-24 10:40
10:40:36.525 [main] INFO org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/ort/eclipsefdn-ort/conf/ort.conf'.
[SNIP]
```
baseURL = "https://ort-vm1.eclipse.org"
#languageCode = "en"
#DefaultContentLanguage = "en"
title = "Eclipse ORT"
paginate = 50
# theme as hugo module
#theme = "github.com/lxndrblz/anatole"
# theme as git submodule
theme = "anatole"
summarylength = 50
enableEmoji = true
enableRobotsTXT = true
timeout = 60000
# Google Analytics
#googleAnalytics = "UA-123-45"
# Syntax highlighting
pygmentsUseClasses = true
pygmentsCodeFences = true
pygmentsCodefencesGuessSyntax = true
[markup]
[markup.goldmark]
[markup.goldmark.renderer]
unsafe=true
[taxonomies]
category = "categories"
series = "series"
tag = "tags"
#[en]
#title = "My blog"
#weight = 1
#LanguageName = "EN"
#contentDir = "content/english"
[[main]]
name = "Home"
identifier = "home"
weight = 100
url = "/"
[[main]]
name = "Projects"
weight = 200
identifier = "projects"
url = "/projects/"
[[main]]
name = "Runs"
weight = 300
identifier = "runs"
url = "/runs/"
[[main]]
name = "Errors"
weight = 400
identifier = "errors"
url = "/errors/"
[[footer]]
name = "imprint"
weight = 1
identifier = "imprint"
url = "/imprint/"
title = "Eclipse ORT"
author = "Eclipse Foundation"
#copyright = "2022"
description = "Eclipse ORT results"
profilePicture = "images/profile.png"
keywords = ""
favicon = "favicons/"
# example ["css/custom.css"]
customCss = []
# example ["js/custom.js"]
customJs = []
images = ["images/site-feature-image.png"]
doNotLoadAnimations = false
# Form Spree Contact Form
#contactFormAction = "https://formspree.io/f/your-form-hash-here"
# Google Fonts
#googleFonts = ["Indie+Flower", "Roboto:ital,wght@0,100;0,400;0,700;1,400"]
singleDateFormat = "2006-01-02 15:04"
indexDateFormat = "2006-01-02 15:04"
listDateFormat = "2006-01-02 15:04"
noreadingtime = true
# Read More links for truncated summaries
# readMore = true
postSectionName = "projects"
reversepagetitle = true # When set to 'true', the Window Title will be reversed to 'Title | Author' instead of the default 'Author | Title'
## Math settings
[math]
enable = false # options: true, false. Enable math support globally, default: false. You can always enable math on per page.
use = "katex" # options: "katex", "mathjax". default is "katex".
## Social links
# use 'fab' when brand icons, use 'fas' when standard solid icons.
[[socialIcons]]
icon = "fab fa-gitlab"
title = "GitLab"
url = "https://gitlab.eclipse.org/eclipsefdn/emo-team/eclipsefdn-ort"
[[socialIcons]]
icon = "fas fa-home"
title = "Home"
url = "https://www.eclipse.org/"
[[socialIcons]]
icon = "fas fa-envelope"
title = "e-mail"
url = "mailto:boris.baldassari@eclipse-foundation.org"
---
title: "ORT Results"
type: list
---
This is a very basic website to display ORT results.
### List of last runs
----
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment