SWT native builds often blocked by aggregation-build test execution

Summary

In the Eclipse RelEng JIPP the build of the native SWT binaries as well as the execution of the platform-specific test for Eclipse SDK I-builds and P/Y-builds all run partly on the special nodes/machines of a specific platform and architecture. The current configuration of these special nodes currently only permits one executor per node, which means that for example only one job at the time can run on the windows or macos-aarch64 machine.

While the part to build SWT native binaries for a specific platform usually runs in a minute or less, the execution of the I/Y/P-builds often runs for about three hours and therefore blocks the special builds in that time. For that reason the SWT pipeline already has a timeout of three hours, while it can complete in under 10min. This can be annoying for change verification builds and if a necessary native-binaries build following a change on the master is blocked for too long. Then other developers can't fetch the new natives or an I-build could start in the meantime and the natives could not match the java code (although this scenario is unlikely because short before an I-build the special build nodes should not be occupied).

Would it be possible to somehow enable SWT natives build in parallel to the I/Y/P-build test execution?

In general I could image the following solutions:

• Pause long running jobs and resume after quick ones are done (don't know a way to implement that)
• Add a second executor for the special platform-specific nodes. But only for fast running jobs, not for long ones in case I/Y/P-builds tests are delayed for some reason and have queued up.
• Move the eclipse.platform.swt project/job back to platform JIPP and add the platform-specific executors there. This is effectively a second executor but only for the SWT natives build, so that that I/Y/P-build tests cannot run at the same time.

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

added IT-severitylow label

added IT-prioritylow label

added service:jenkins statetodo team:releng labels

Does the SWT natives build include any UI tests?

The build of the natives itself does not include any UI tests. It just compiles the native sources and stashes the built native binaries.

The tests only run later as part of the Maven build stage which runs on the ordinary a centos-latest agent.

I see two possible solutions:

• Use a second executor. In order to avoid multiple long-running I/Y/P builds blocking both executors, the "disable concurrent build" option should be activated. If that is not enough, we could install the lockable resources plugin to tune this even more.
• Create a second agent pointing to the same machine. This would make it easier to separate jobs, but comes with the overhead of configuring two agents for the same machine.

Using disable concurrent build will probably not work, because AFAIK this only affects concurrent builds of the same job, but the lockable resources plugin looks promising (I didn't knew this exists).

In the end I think both solutions would work. With your first option the I/Y/P build tests would have to be adjusted to use the locks (in the end it would probably be me who has to adjust them). With a second agent for each of the special machines this would not be necessary. And, if I'm not mistaken, would have the advantage that the eclipse.platform.swt job can be moved back to the platform Jenkins were it originally belonged to, can't it? There is also some discussion/consideration to also build the equinox/eclipse launcher binaries as part of its Jenkins pipeline (there is no dedicated issue for that yet) and with the current configuration this would probably mean that the equinox-job would have to move to the releng-Jenkins too, wouldn't it.

On the other hand configuring two (or maybe three, if equinox also goes this path) agents would probably mean more work on your end, but I cannot assess how much and which approach is less work in total. Personally I would be in favour of option two, but I'm of coursed biased since it is less work for me, but option one should work as well.

For the migration of the Equinox launcher+executable builds I have now created issue equinox#575.

If own agents for the special, platform-specific build machines are added to the platform-JIPP we should do the same for equinox as well in order to keep it consistent.

I have no experience and knowledge in managing multiple Jenkins instances together but would it be possible to have some sort of abstract definition or template or similar for agents on a machine that can defined in a superordinate 'controller' of multiple instances, so that you can just say that machine should have an agent for Jenkins1, Jenkins2 and `Jenkins5?

assigned to @fgurr

changed the description

mentioned in issue #4609 (closed)

In a (private) chat you mentioned that it would maybe be sufficient to copy the jenkins/configuration.yml of the eclipse.platform.releng JIPP to the eclipse.platform and eclipse.equinox JIPP, if the workspace paths were adjusted. I thought about this a bit more and instead of copying the configuration, which means it has to be kept in sync, we could just sym-link the configuration into the other JIPPs that should use the same static agents/nodes. With that in mind I have now created a PR with suggestions, hoping that it helps to make the static build agents available in the eclipse.platform and equinox JIPP in an elegant way:

• https://github.com/eclipse-cbi/jiro/pull/379

mentioned in issue eclipse-wg/ide-wg/ide-wg-dev-funded-efforts/ide-wg-dev-funded-program-planning-council-top-issues#36

mentioned in issue #4832 (closed)

marked this issue as related to #4609 (closed)

Just as a note so that it's not forgotten:

• We don't need all native agents from the releng JIPP in platform and equinox. It is sufficient to have one agent for each distinct native.builder-*** label.
• In case you want to prioritize parts of this work: having the agents in the platform JIPP would be the greater benefit because SWT binaries are built more often than Equinox launchers.

added wg:Eclipse IDE label

To avoid having several agents in a disconnected/offline state attached to the Equinox and Platform JIPPs, I've removed all static agents for now. I will re-add them, once they are set up correctly.

This hopefully avoids some confusion for both committers and EF IT colleagues.

The following Jenkins agents connected to the Eclipse Platform Releng JIPP use the swtbuild user:

• ppcle-buildTest
• ppcle-test
• centos-aarch64-1

@gpunathil @rahulmoh in order to connect the same agents also to the Platform and Equinox JIPPs, please set up new Jenkins bot accounts on these machines (e.g. swtbuild-platform and swtbuild-equinox with different credentials). If this is done, please share the credentials with us safely, e.g. as described here: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Sharing-secrets.

/cc @hwellmannwr6 @slakkimsetti

please set up new Jenkins bot accounts on these machines (e.g. swtbuild-platform and swtbuild-equinox with different credentials).

Just a nit-pick, but maybe more general names like eclipse-platform-builder and eclipse-equinox-builder or -bot would suite a bit better?

As long as the project short name appears in the username, I'm fine with it. Shorter names are always preferred (e.g. just platform or eclipse-platform).

@gpunathil @rahulmoh any update here?

Created account swtbuild-platform and swtbuild-equinox

Thanks, Rahul!

Is there any progress on this issue in the meantime?