Get more processing power on GH
Summary
GH actions "free" has several processing limits see
as we use more and more from GH this becomes a bottle-neck (especially for the "eclipse" organization that hosts a lot of projects), I therefore like to start the process of evaluate if/how we could get more processing power, this might include:
- Choosing an 'Enterprise' Plan, maybe even a custom contract
- Adding EF hosted runners see https://docs.github.com/en/enterprise-cloud@latest/actions/hosting-your-own-runners/about-self-hosted-runners#usage-limits
- a combination of both
I think especially the opportunity for adding windows+mac runs is very useful as this was always problematic/limited on the Jenkins Infrastructure.
FYI @emerks @akurtakov @aloskutov @mistria ... and maybe more ...
Activity
added IT-prioritylow IT-severitylow labels
I think especially the opportunity for adding windows+mac runs is very useful as this was always problematic/limited on the Jenkins Infrastructure.
- Can you elaborate what exactly is/was problematic/limited?
- Are you suggesting to use GitHub-hosted runners for MacOS and Windows? Or EF hosted runners?
Can you elaborate what exactly is/was problematic/limited?
Whenever it was asked for such things, I need to find a strategic member paying thousands of $$$ per year to get an executor assigned to this project. GH gracefully offers this for free (on a limited basis of course). I'm even attached a selfhosted node for Tycho already because it is much "cheaper" and easier than the "official" way of getting more runners assigned...
Are you suggesting to use GitHub-hosted runners for MacOS and Windows? Or EF hosted runners?
Github already allows linux/win/mac runners, but you have to pay for getting more runs in parallel, as an alternative you can connect self-hosted runners (either linux, mac or windows nodes).
So if EF chooses it is cheaper to pay GH that's fine, if we think we can do better EF can provide additional EF hosted Github runners.
By the way I can understand that this all is "very low IT priority", but from a contributors perspective this is really an serve issue if I have to wait hours to get my contribution checked and could make you really mad... so I really would appreciate a label like "Improve contribution experience" or something like that.
assigned to @fgurr
-
IIUC, the main issue is the concurrency limit of 20 jobs per GitHub organization. At the moment I see between 12 and 16 jobs running in the eclipse org at the same time, so the headroom is limited.
We can work around this by moving project repos to dedicated GitHub organizations. With a +1 from the Tycho project lead we can create https://github.com/eclipse-tycho and move the Tycho repo there. Once Tycho alone exhausts the concurrency limit we can assist with figuring out a way to upgrade to a paid/sponsored plan.
-
While this would work for tycho (WDTY @akurtakov @mistria ) I think this is a bit circumventing the limits and EF really should check if it would be worth to enter a contract with Github I think as it seems a bit cumbersume to move things to a different org for the whole purpose of getting more resources (even though its not forbidden by Gh I think ...)
-
We have been recommending a "one GitHub organization per project" strategy for quite some time (not only to work around concurrency limits).
Moving projects to dedicated orgs can be done on demand, so churn is limited.
The "enter a contract with GitHub" discussion is something that we will pursue as well, but I don't expect this to happen in a matter of days or weeks.
-
@fgurr what makes it a "good time", e.g. should we ask contributors not to push any changes fro a while? If yes, please just let us know a timeframe that fits you best and I'll announce that on the dev mailinglist.
-
If there are no releases/milestones planned in the coming days it should be fine. I'd estimate the disruption to last a few hours.
During the transfer committers will lose access until the sync script fixes the permissions (~2-4h). Local dev environments and CI configurations will need to be adjusted.
I'd propose tomorrow, August 9, 10:00 (AM) CEST.
+1 for announcing this on the dev mailinglist.
-
Announcement is out: https://www.eclipse.org/lists/tycho-dev/msg02103.html
-
Transfer is done. => https://github.com/eclipse-tycho/tycho
Now we are waiting for the sync script to catch up.
-
Thanks that was really fast and smooth! It seems that there is even a redirect from https://github.com/eclipse/tycho/ -> https://github.com/eclipse-tycho/tycho
