Skip to content

Check as much as possible during a build, don't stop on first issue

Origin and goal

This originally was discussed at !512 (comment 1109476), point 4d:

d) @ahofkamp and @ddennis agree that if the bulid enforces certain things, it would be better to report any non-critical things (that don't cause compile errors for instance) at the end of the build. This allows as much of the build to proceed as possible, and as much of the issues to be reported (and thus fixed) at once, allowing the issues to be fixed with less iterations. This applies to the formatter profile, JDT checks, Checkstyle checks, AsciiDoc checks, test failures, license header checks, etc. The proposal is to make this possible.

The goal is thus to check more during a build, fail only at the end, and report all problems in a single report. It is not the goal to add more checking during the build, remove checking, or change checking. It is only about reporting found issues at the end.

What do we check?

Currently, during the build, we check:

Artifacts Plugin/script Checks
Documentation AsciiDoctor Invalid syntax, missing attributes, missing images, etc.
Documentation AsciiDoc Source Checker (org.eclipse.escet.common.asciidoc.source.checker.AsciiDocSourceCheckerMojo) Single sentence per line, source block indent, etc.
License headers Custom script (misc/license-header/license-header-check.bash) Missing/wrong license headers.
Sources Tycho compiler Various compiler and static checks.
Sources Checkstyle Various static checks.
Tests Tycho surefire Failed/crashing tests.

We don't currently check for formatting (#534) and third party dependency licenses (#360).

First ideas

Some first ideas were given at !512 (comment 1109488):

Regarding 4.d:

  • Maven has the -fae / --fail-at-end option to not fail the build on the first plugin with test failures, but to continue, and report them only at the end. See https://maven.apache.org/ref/3.6.3/maven-embedder/cli.html.
  • The Checkstyle Maven plugin has various options to only check and not report, check and report, report on what was already checked, report aggregate violations in a report, etc. See https://maven.apache.org/plugins/maven-checkstyle-plugin/plugin-info.html.
  • If some plugins don't support this natively, we could configure them to report only warnings and we could collect the warnings ourselves at the end of the build. We could do this by running the Maven build and teeing the output to a file, and getting the warnings from that.

For 4.d, the Maven 'reporting' functionality may also be of interest: https://maven.apache.org/plugins/maven-site-plugin/examples/configuring-reports.html

Current ideas

Maven has the -fae / --fail-at-end option. See https://maven.apache.org/ref/3.6.3/maven-embedder/cli.html. This ensures Maven does not fail on the first plugin with test failures, but continues, and reports test failures at the end. However, likely we can't combine this with other types of things to report. Hence, we likely can't use it. Instead, I propose to look if we can can configure what happens for test failures in the Tycho surefire plugin (only test but don't report, or report but don't fail, or so).

The Checkstyle Maven plugin currently fails on first plugin with Checkstyle issues. The plugin has various options to only check and not report, check and report, report on what was already checked, report aggregate violations in a report, etc. See https://maven.apache.org/plugins/maven-checkstyle-plugin/plugin-info.html. Likely we could configure it only check and report, but not fail. We can collect the problems/report at the end ourselves.

Likely some of the other plugins have similar options. We should check each plugin for what it exactly supports.

We would have to make a plugin that somehow is always at the end of the build sequence (it depends on all other projects, or we can minimize it by making use of recursive dependencies, as the product and test aggregation depend on most other things already). It collects all issues reported by the various plugins for the various projects that we build. And we generate a single report from that. We make the build fail then as well, if there are any issues. We should also attach the generated report as build artifact in the Jenkinsfile then.

Rather than doing everything ourselves, we could also check the Maven 'reporting' functionality. Maybe we can reuse it? See https://maven.apache.org/plugins/maven-site-plugin/examples/configuring-reports.html.

Steps

Edited by Dennis Hendriks

Copyright © Eclipse Foundation, Inc. All Rights Reserved.     Privacy Policy | Terms of Use | Copyright Agent