Improve the resources available to ci.eclipse.org instances

Summary

There has been criticism that GitHub runners offer better resource for running GitHub actions than ci.eclipse.org offers for running Jenkins jobs.

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

During discussions this week in Montreal it was suggested that I should open an issue about this topic, so here it is.

I don't claim to know what the best solution or compromise is for this issue. Of course many if not most projects function just fine within the current constraints and allocating more resource is obviously not without cost.

Steps to reproduce

Any job by default can use at most 4g, although it's possible to configure a job that uses more. That being said, I believe the executor can use at most 8g in total so such an increase for one job would limit the executor to running just a single job on the one executor allocated to the ci instance.

What is the current bug behavior?

We've seen many jobs being killed for exceeding the quota. I'm not sure what has changed to make that problem worse than a few months ago. Often running the same job again then works. It's a source is significant frustration and is also bad for making resource consumption worse.

What is the expected correct behavior?

It should be easy/easier to configure a job that uses more memory. Also, ci.eclipse.org would be seen as more competitive if executors could use 16g like GitHub runners.

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

I think some projects would benefit from improved resource availability.

added IT-prioritylow IT-severitylow service:CI service:jenkins team:releng labels

added statetodo label

Thanks for opening this. We have not evaluated the current default offering for CBI in a long time, to see how it stacks up against what's out there. It will be a good and worthwhile exercise.

assigned to @fgurr and @droy

mentioned in issue #4556 (closed)

mentioned in issue eclipse-wg/ide-wg/ide-wg-dev-funded-efforts/ide-wg-dev-funded-program-planning-council-top-issues#15

I wonder if there is some statistics about the overall usage of RAM/CPU over some days?

On which level? Individual Jenkins instance or cluster level? What insight are you expecting?

I'm not sure if the two most recently failures of https://ci.eclipse.org/oomph/job/build/job/master/ are memory issues but I can just say such failures drive me crazy:

[INFO] [main] + tar --format=gnu -cf ../org.eclipse.oomph.setup.installer.product.with-jre.restricted-win32.win32.x86_64.tar artifacts.xml configuration eclipse-instc.exe eclipse-inst.exe eclipse-inst.ini features p2 plugins readme /home/jenkins/agent/workspace/build_master@tmp/durable-e29e2dfc/script.sh.copy: line 2: 200 Killed mvn clean verify --no-transfer-progress -DskipTests=

SKIP_TESTS -Dmaven.repo.local=

WORKSPACE/.m2/repository -Dorg.eclipse.justj.p2.manager.build.url=

JOB_URL -Dbuild.type=

BUILD_TYPE -DRESTRICTED_VERSION=

RESTRICTED_VERSION -DPACK_AND_SIGN=

SIGN -Dbuild.id=

BUILD_NUMBER -Dgit.commit=

GIT_COMMIT

What can I do to determine why this was killed?

Please create a helpdesk issue so we can investigate.

The next time it happens again, I will create a dedicated issue.

Adding context:

When JIRO was created ~5 years ago, the default resources were based on what GitHub provided by default for free for public projects at that time (Linux OS, 2vCPUs, 7GB RAM and 14GB of SSD storage). The default configuration for JIRO pods was set to 4GB of RAM to allow two parallel builds as a compromise between single build resources and parallelism. The resource pack concept gives projects the flexibility to add more resources if required.

I've tried to find an official announcement about the resource changes for GitHub runners, but at least the GitHub blog (https://github.blog/changelog/label/actions/) did not seem to mention it. The GitHub runner documentation was changed in January 2024 according to this GitHub commit: https://github.com/github/docs/commit/6aa6bc552ed9cbb2cdb6529892d188b26cac9805#diff-143e31b92db0cd84bb148b7f67795481af59b4c13cb56c312ab0cb923219bd86R24

GitHub's new default resources are:

• Linux OS (Ubuntu LTS)
• 4 CPUs
• 16GB RAM
• 150GB SSD storage

We've seen many jobs being killed for exceeding the quota. I'm not sure what has changed to make that problem worse than a few months ago. Often running the same job again then works. It's a source is significant frustration and is also bad for making resource consumption worse.

Some thoughts:

• IMHO, the default resource offering is still good for 80%+ of our projects.
• 20% of the projects have higher resource demands. We need to improve our offering for them.
• 5% of the 20% will always have higher demands.
• Jobs being killed for exceeding the quota can have many reasons. One job simply exceeding the max amount of RAM is just of them. In many cases, pods not being cleaned up correctly, pods being run at the same time or general cluster issues can lead to killed jobs. This needs to be investigated.
• It's not easy to see a trend that quota issues became worse in the last months, other than hearing it from projects. We are working on improving our monitoring solutions to analyze such trends and be alerted early.
• There were a few instances where memory quotas ultimately lead to identifying memory leaks in the code. I assume that higher quotas will keep them hidden for longer, but I realize that I won't be able to stop this and will resort to yelling at the cloud.

It should be easy/easier to configure a job that uses more memory.

No doubt. We will discuss how we can make this easier to handle. E.g.

• creating default pod templates with more memory for all JIPPs (we already added modified pod templates for specific projects that ran into memory issues, like Oomph, see https://github.com/eclipse-cbi/jiro/blob/master/instances/tools.oomph/config.jsonnet#L19)
• add better documentation how memory can be increased with custom pod templates
• reconsider the upper memory limit for a single build job

Also, ci.eclipse.org would be seen as more competitive if executors could use 16g like GitHub runners.

While certainly true, I doubt that we will be able to compete with GitHub runners. We can try to play catch up, but eventually the abundance of free build resources is just one of the ingredients of the developer's crack that lures devs and projects to move to GitHub. The hangover is far away and/or well hidden.

Considering the JIPPs stability, isn't it best to allow publishing to download.eclipse.org from GH actions? That way projects will not be blocked by issues.

Considering the JIPPs stability, isn't it best to allow publishing to download.eclipse.org from GH actions? That way projects will not be blocked by issues.

Also code-signing / pgp is only available on Jenkins builds afaik