Random build failures of the GlassFish project

Summary

For some 2 weeks we experience instability of GlassFish builds.

Note: The build first compiles and installs all artifacts without a test execution. Tests are executed later.

Steps to reproduce

Create a PR or rerun some build.

What is the current bug behavior?

• Build starts - sometimes passes, sometimes fails. Every time in different part(s) (some are more probable, some less)
• Once it crashed while it was cloning a github repo
• Several times in Ant tests
• Several times in Maven tests
• Several times while compiling sources or generating javadoc jar file.
• Builds are now also slower, and I believe it is not caused by changes in the project as we don't see it in local builds.

What is the expected correct behavior?

Reproducible builds.

Relevant logs and/or screenshots

• https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/job/PR-24251/6/execution/node/193/log/

• https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24251/7/execution/node/111/log/

  • both simply vanished in the middle of the build

• https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24218/1/execution/node/299/log/

  • 2 runs failed, third passed: https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24218/

• https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24247/

  • build6 https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24247/6/execution/node/159/log/
    • test failure - instance in a cluster did not make it in time (this did not happen until the end of 2022)
  • build5 - same
  • build4 was aborted
  • build3
    • weird failure of the logging tests unrelated to the change in the PR. Note that logging is built as one of first modules, all other depend on it: https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/job/PR-24247/3/execution/node/166/log/
    • 4 failures in QuickLook tests are alright, they really detected an error.

• etc., see https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/view/change-requests/

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

• We have to rerun the build several times for nearly every PR
• Builds are not reproducible - every time crashes different part.

Related Questions

• Last year OmniFish joined Eclipse Foundation and we assigned our resource packs to GlassFish.
• How many resource packs is currently assigned to GlassFish?
• Where can I find some records about it or even be notified about changes and possible consequences?
• Did something change in January 2023?
• What else? Is there something we could do?