Improve Jenkins / Jiro CBI build stability

Jenkins and Nexus stability have been quite poor recently. The root cause is a storage device that is beyond its capacity. The EF Releng team has cleared the roadblocks that were blocking the migration to a more stable, robust and scalable storage, and has already migrated 21 of the 240 instances.

This issue is to communicate that the team is aware of the issues, and to communicate progress.

[███████████████████████████████████████████████████████████████████████████████████████████████-95%-]

added Blocker critical team:infra team:releng labels

assigned to @droy and @fgurr

changed the description

added service:jenkins label

changed the description

mentioned in issue #3745 (closed)

We reached 20% (49 migrated JIPPs) today.

To set expectations: while this is good progress, the amount of low-hanging fruits is limited and will result in (a lot slower) progress towards the end. The storage device will be a huge bottleneck during the migration of larger instances.

changed the description

Not surprisingly, two of the biggest CI instances (JakartaEE-TCK and Glassfish) were clearly identified to cause a lot of load and traffic on our storage server, whenever excessive builds are started. There might be others as well. The plan is to move those two to CephFS ASAP.

/cc @emerks

mentioned in issue #5070

marked this issue as related to #5070

mentioned in issue #5072 (closed)

marked this issue as related to #5072 (closed)

Please migrate repo.eclipse.org with priority (when it's down all jipps are more or less useless). Related to #5130 (comment 2836443)

changed the description

The releng team has crossed the 35% mark, and some larger instances have moved over, leading to better stability for all instances.

Please migrate repo.eclipse.org with priority

This is the plan; there are specific challenges to contend with. At this time, stability has much improved already, and the team is not even halfway done.

Thanks for your patience.

changed the description

50% of the Jenkins instances have been migrated.

changed the description

At this time, we've crossed the 62% mark, and although we're not done, the CI service is no longer considered 'degraded'. More to come.

Please migrate repo.eclipse.org with priority

That work is scheduleed for Nov 12 at 09:00pm CET https://www.eclipsestatus.io/maintenance/456545

Reminder: this happens tonight at 09:00pm CET!

How can specific projects know whether their Jenkins instance has been migrated already?

They can't and should not need to, since it's a transparent change. In some cases, we ask the project to support clean up tasks before the migration, though.

We're seeing some weird errors in the Eclipse ESCET builds we've not seen before, and we're wondering if it is due to not yet having been moved, or maybe because we were moved and this caused unexpected changes somehow.

The ESCET Jenkins instance has not been moved yet. Please create a separate HelpDesk issue and point us to the errors so we can take a look.

repo.eclipse.org has been moved to the new backend storage device.

changed the description

75% of the Jenkins instances have been migrated (182 of 240).

changed the description

85% of the Jenkins instances have been migrated (205 of 239).

changed the description

95% of the Jenkins instances have been migrated (11 JIPPs to go).