Add script to compare outputs of Maven builds
When you change something that affects the build, it can be useful to compare the output of the build before and after the change. For instance, in #569 (closed) we migrate from JUnit 4 to JUnit 5. It is then good to check that the same tests are still being executed. Also, when upgrading Eclipse, Maven, Tycho, etc, it is good ensure that we are not introducing regressions in the build. What if the build still succeeds, but no products are being built or no tests are being run. Clearly, that would not be desired.
Comparing the build output of develop
and branch or merge request is however not always as simple as you'd like it to be, as the output may differ for non-important reasons. There are multiple reasons for this, and some examples include:
- The branch name is used as a directory during the build and thus pops up here and there in the build output.
- The qualifiers of plugins include the date/time of the last change and thus typically differ between builds after a change was done.
- For tests, the time it took to execute tests is indicated, and this generally differs slightly between builds.
Hence, it is useful to have a script that allows you to compare build outputs more easily, but first getting rid of the irrelevant differences, and then doing the comparison. That way, finding relevant differences (e.g., regressions), is much easier in the full diff of the build outputs. I've made such a script locally, and I've been using it for a while.
Obviously, if your intent is to change something to the build output that is normally considered irrelevant, then first getting rid of those differences will not work. The script is thus meant to work in most cases, not all cases. Still, my experience over the last few months has shown this to be a useful script in practice.