OSEE Migration to gitlab or github

changed the description

added IT-prioritylow IT-severitylow Migration ~16083 labels

assigned to @fgurr

First, how long will Jenkins be available before it is retired?

We have no plans to retire Jenkins.

Second we have heard Eclipseᵀᴹ will not provide any GitLab runners and we will need to use our own hosted runners. Is that true?

We are planning to make GitLab runners available for all projects later this year.

Finally, if we instead decide to move to GitHub, will we be permitted to leverage the existing Actions CI/CD infrastructure that is available to public projects there?

Yes. Please note, that some services (like signing) are only accessible from inside the EF network for now.

Hello Frederic!

Thank you for getting back to use so quickly!

As a followup, we have two additional questions:

What will the specifications and runtime limits of the supplied GitLab runners be?
Is there a ticket or information source we can follow to stay up to date on the latest developments regarding GitLab Runners and the migration out of Gerrit as a whole?

As a closing note, our project lead (Don Dunne) has expressed interest in making use of the runners as soon as they are available. If there is any need for projects to beta test, we would be happy to do so.

What will the specifications and runtime limits of the supplied GitLab runners be?

Specs will be similar to a resource pack that is provided by default for every project's Jenkins instance.
See https://wiki.eclipse.org/CBI#What.27s_provided.3F (tl;dr: 2vCPUs, 8GB of RAM)

Is there a ticket or information source we can follow to stay up to date on the latest developments regarding GitLab Runners and the migration out of Gerrit as a whole?

We will announce changes on the cross-project-issues-dev mailing list:
- https://www.eclipse.org/lists/cross-project-issues-dev/index.html
Migration plan:
- https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Gerrit/Gerrit-and-Bugzilla-deprecation-and-migration-plan
Tickets related to Gerrit shutdown:
- #680

As a closing note, our project lead (Don Dunne) has expressed interest in making use of the runners as soon as they are available. If there is any need for projects to beta test, we would be happy to do so.

Thanks. We will put you on the GitLab runners guinea-pig shortlist.

@ddunne @broberts3cp @grumble Do you have a timeline for when you want to do the migration to GitLab?

Hello Frederic,

That largely depends on the state of GitLab runners. I have not been keeping up and just skimmed over some of the links you sent previously. I did not see anything related to GitLab runners in my cursory review. Have they been made available yet or have a concrete timeline to be made available?

Ideally, we would migrate once runners have been made available to avoid additional work to implement a Jenkins integration and later have to change to runners.

Have they been made available yet or have a concrete timeline to be made available?

There is no timeline for when GitLab runners will be available. We made some good progress, but we have a few blocking issues to resolve before projects can test them.

Ideally, we would migrate once runners have been made available to avoid additional work to implement a Jenkins integration and later have to change to runners.

I expect the required changes in your Jenkins configuration for the migration from Gerrit to GitLab to be minimal (basically changing the repo URLs). In contrast, setting up GitLab CI pipelines from scratch will be a lot more work. So I'd recommend not waiting on the availability of GitLab runners.

I see. Fortunately, the GitLab pipeline setup would not be from scratch given I have already created functioning GitLab jobs for most of our use cases.

How likely is the GitLab runner work to be completed prior to a forced migration?

In any case, I will discuss with the team and get back to you.

How likely is the GitLab runner work to be completed prior to a forced migration?

I don't know OSEE's position in the queue for "forced migrations" (more like a strong recommendation with a reasonable waiting period), so I can't comment on the probability that GitLab runners are available by then.

I have just had a short discussion with the team.

The permissions scheme for GitLab was one of the things that dominated the conversation.

From what I have read: Default permissions:

Project leads get "Maintainer" permissions.
Committers of a project get "Developer" permissions.
Contributors can get "Reporter" permissions.

This scheme would introduce significant pain points for our team. I am aware we cannot "Add new team members", but are we permitted to modify the permissions of users without changing the eclipse team structure i.e. having to elect additional project leads and committers? If we do have to elect additional project leads and committers, how much of an endeavor would that be?

This scheme would introduce significant pain points for our team.

Can you elaborate on these pain points and how this is different from the current permission scheme?

are we permitted to modify the permissions of users without changing the eclipse team structure i.e. having to elect additional project leads and committers?

No.

If we do have to elect additional project leads and committers, how much of an endeavor would that be?

See https://www.eclipse.org/projects/handbook/#elections-committer and https://www.eclipse.org/projects/handbook/#elections-pl for more information about the process.

The primary pain point arises from the setup, modification, and maintenance of GitLab CI pipelines. With the way the GitLab permissions scheme is setup and how it is paired with the eclipse team structure, it limits many basic and essential actions to project leads i.e. Don and Ryan. For example, "Manage CI/CD settings", "Manage job triggers", and "Manage project-level CI/CD variables" are reserved exclusively to Project Leads. Given Ryan is taking more of a backseat these days, that leaves everything to be performed by Don, hence the pain point. We have already had committers try to experiment with GitLab CI and become frustrated because there is much they cannot do.

All this said, I can understand the opposing view that this level of restriction is important for maintaining the security of the project and I can appreciate that view point for an open source project especially. My intention is not to argue, but only to inform.

I will take what I have learned here and work with the team to devise a game plain for migration that will hopefully mollify any pain points. I will get back to you with what we decide after a meeting next week.

Thank you for your time, Frederic. I appreciate the thoroughness and promptness of your responses.

We have already had committers try to experiment with GitLab CI and become frustrated because there is much they cannot do.

This might make the case for staying with Jenkins CI for the time being if it gives the team more flexibility.

After a meeting with the team, it seems that prior to migrating we will need to hold a series of elections. We currently have too many core members that are not committers. Under the GitLab permissions scheme any non-committer will have to jump through hoops we are not accustomed to when contributing, which would likely slow our development considerably. Once the issues with our eclipse team composition have been addressed, I will have a better estimate for you on our migration timeframe.

Hello, Frederic.

A question recently came up that I'd like some context on if possible. It has been my understanding that to be able to delete a branch you need to prepend the branch name with your username. Presumably the same holds true for tags. I have never attempted otherwise for fear of creating an unnecessary and persisting branch, so I don't know this to be true by experience. Is there documentation on this somewhere? Furthermore, assuming I am not entirely off base, will branches and tags in GitLab also have these limitations? This will help inform how we modernize our software development process when we migrate.

It has been my understanding that to be able to delete a branch you need to prepend the branch name with your username.

This has been a best-practice that we have used on Gerrit to allow more self-service for committers and avoid many "I have accidentally created a branch, can you please delete it for me" HelpDesk tickets.

AFAIK, this can't be replicated on GitLab since permissions are not as granular as on Gerrit. Committers have "Developer" permissions, which gives them the power to delete non-protected branches and tags (https://docs.gitlab.com/ee/user/permissions.html). We recommend using protected branches (e.g. at least the main/master branch). Protected branches can only be handled by project leads.

Any news about the timeframe for the migration?

Hello Frederic,

The team had actions planned for the migration in this project increment, but some other high priority actions shook up our plans. We would like to be on GitLab three months from now, but while we are still in flux, I unfortunately can not make any guarantees. I brought this issue up again in today's meeting and will continue to keep it present in our team's conversations.

Good afternoon, Frederic.

It has been a long road, but it seems that runners are finally available in Eclipse GitLab. Unfortunately, it does not appear that the resource allotments will meet our needs.

If I understand correctly, an open source project can exist publicly in GitHub and utilize the resources they offer to opensource projects, all while maintaining their relationship with eclipse. Can the same be said of GitLab?

Frederic:

We are very interested in staying as an open source eclipse.org project but need to have the resources to continue with GitLab. We unfortunately are not able to get Boeing to contribute at this time. Is there a way to get an exception to the resource pack allotment for the associate/contributing membership level?

Don

If I understand correctly, an open source project can exist publicly in GitHub and utilize the resources they offer to opensource projects, all while maintaining their relationship with eclipse. Can the same be said of GitLab?

Similar to GitHub, projects on https://gitlab.eclipse.org exist publicly and can utilize the resources that are offered to open-source projects.

There is a big difference between GitHub and our GitLab instance, though:
GitHub is a cloud offering, and by the grace of GitHub, you have access to "free" compute resources. On the contrary, we host our GitLab instance (https://gitlab.eclipse.org) and compute resources on our own infrastructure. It's not hosted at https://gitlab.com (which would be a better comparison to https://github.com).

We are very interested in staying as an open source eclipse.org project but need to have the resources to continue with GitLab.

Can you provide an estimate about the required resources that you would need?

We unfortunately are not able to get Boeing to contribute at this time. Is there a way to get an exception to the resource pack allotment for the associate/contributing membership level?

You can see resource pack allotment here: https://github.com/eclipse-cbi/cbi/wiki#resource-packs-included-in-membership

It's not hosted at https://gitlab.com (which would be a better comparison to https://github.com).

Yes, I probably was not clear, but https://gitlab.com is what I was referring to. So for example, if we were to join the "GitLab for Open Source Program" would we still be able to maintain our Eclipse membership? Not to say that is the route we want to take, but it would be able to better support or compute requirements.

Can you provide an estimate about the required resources that you would need?

The most constraining factor is memory. In both our web testing and java testing we are already hitting the limit on the resources made available in Jenkins. Most likely we will need greater than 8gb of memory for our Java jobs and greater than 4gb of memory for out angular jobs.

So for example, if we were to join the "GitLab for Open Source Program" would we still be able to maintain our Eclipse membership?

@wbeaton can you comment on this?

The most constraining factor is memory.

Without any additional resource packs, you can run builds with up to 8GB of RAM, but you will be limited to one build job at a time. In order to utilize more than 8GB of RAM during builds, you would need to get sponsorship for one additional resource packs by any eligible member organization.

Without any additional resource packs, you can run builds with up to 8GB of RAM, but you will be limited to one build job at a time.

Oh, I was unaware of that. How does that work? I thought the documentation listed that only the large runner and the dedicated agent had 8gb of RAM, but neither were available at 1 resource pack. Don't get me wrong, I am ecstatic to hear that, but I am curious if I am misreading the docs or missing a section.

So for example, if we were to join the "GitLab for Open Source Program" would we still be able to maintain our Eclipse membership?

Membership means something specific at the Eclipse Foundation. What I assume you mean is whether or not you can operate as an Eclipse open source project.

We require that Eclipse open source projects use Git repositories that are managed by the Eclipse Foundation. We do not currently have a means of managing Git repositories on gitlab.com, so the answer is no.

We will take that into account in our decision making process. Thank you for the info, Wayne.

Oh, I was unaware of that. How does that work? I thought the documentation listed that only the large runner and the dedicated agent had 8gb of RAM, but neither were available at 1 resource pack. Don't get me wrong, I am ecstatic to hear that, but I am curious if I am misreading the docs or missing a section.

The documentation states that a resource pack provides 8GB of RAM in total. By default, the predefined build containers on a Jenkins instance (e.g. basic, centos-7) are specified with 4GB of RAM. This allows to run two build containers at the same time.

The amount of RAM (limits and requests) can be customized in the container spec in a Jenkinsfile (e.g. like here: https://git.eclipse.org/c/osee/org.eclipse.osee.git/tree/plugins/org.eclipse.osee.web/Jenkinsfile?h=dev#n17).

Ah, that is where the disconnect is. Yes, for Jenkins that is accurate. My apologies, I was referring to the runner resources available in Eclipse GitLab, but did not state that clearly enough.

i.e. https://github.com/eclipse-cbi/cbi/wiki#gitlab-runners

At the moment, this is not working.

GitLab runner offers the possibility to override CPU/MEMORY LIMIT/REQUEST per job or pipeline, see: https://docs.gitlab.com/runner/executors/kubernetes/index.html#overwrite-container-resources But I have to set up in GitLab runner definition this new ENV value for each resource: *-overwrite_max_allowed. https://docs.gitlab.com/runner/executors/kubernetes/index.html#cpu-requests-and-limits

Once this is available, you will be able to customize your jobs' memory/CPU settings based on these limits, similar to Jenkins.

The feature is now live.

See a job running on a small runner with more CPU and MEMORY here: https://gitlab.eclipse.org/eclipsefdn/it/releng/gitlab-runner-service/gitlab-ci-templates/-/jobs/350551

The .gitlab-ci.yml configuration:

  variables:
    KUBERNETES_CPU_REQUEST: "3"
    KUBERNETES_CPU_LIMIT: "5"
    KUBERNETES_MEMORY_REQUEST: "5Gi"
    KUBERNETES_MEMORY_LIMIT: "5Gi"

"CPURequest" overwritten with "3"
"MemoryRequest" overwritten with "5Gi"
"CPULimit" overwritten with "5"
"MemoryLimit" overwritten with "5Gi"
Using Kubernetes namespace: grac-cbi

Default value on a small runner:

  build_cpu_request::250,
  build_cpu_limit::500,
  build_mem_request::1024,
  build_mem_limit::1024,

Maximum values are defined by kubernetes resource quotas calculation based on CBI resource pack.

Ah, that's an excellent new feature!

Maximum values are defined by kubernetes resource quotas calculation based on CBI resource pack.

How does that calculation work? For example, given one resource pack, what are the set maximums and how does that play into concurrency?

I have written a new paragraph about that: https://gitlab.eclipse.org/eclipsefdn/it/releng/gitlab-runner-service/gitlab-runner-service-documentation#push-the-limits

Let me know if it's not clear enough.

I like the section title!

Yes, that answers the questions I had. Thank you! I will share this with the team.

I have presented all of the information to the team and there appears to be consensus on migrating to GitHub.

However, there is not consensus on whether we create our own organization or slot ourselves directly under the Eclipse GitHub organization. To help make this decision, can you help identify the differences between the two options? Obviously there are "large runners", but that holds no bearing for us at our membership level.In particular there are two points of interest:

Does creating our own organization provide us any additional freedoms or autonomy to control our repository?
How does the compute resource allocation differ between the two option?
1. Does being under the Eclipse organization still give us access to all the free runners and offerings GitHub provides to public projects as described here without restriction?
2. How does concurrency come into play? By default, public projects get 20 total concurrent jobs and 5 concurrent macOS jobs. However, what happens if we exist under the Eclipse organization? If we slot under the Eclipse organization we will have to share the Enterprise plan's 500 total jobs and 50 macOS jobs with the ~175 other groups (the number may have changed since the email was sent out)? If that is the case, would that mean less than 3 concurrent jobs per group and not even 1 concurrent macOS job (assuming worst case, equally shared loading)? Would we get the full guaranteed 20 concurrent jobs if we made our own organization?

To make this plain and simple. We won't add any new projects to the https://github.com/eclipse organization. We would create a new dedicated GitHub org (in your case https://github.com/eclipse-osee) that is part of the Eclipse GitHub Enterprise account. Creating a new GitHub org independent of the Eclipse GitHub Enterprise account is not an option.

Does creating our own organization provide us any additional freedoms or autonomy to control our repository?

Yes. In your dedicated organization (under the Eclipse GitHub Enterprise account), you will get self-services capabilities (see https://github.com/eclipse-csi/otterdog).

How does the compute resource allocation differ between the two option?

Both the Eclipse org and any dedicated Github org are part of the Eclipse GitHub Enterprise account and share the Enterprise concurrency limits. As a snapshot in time, with ~200 GitHub orgs we utilize less than 50 concurrent jobs at the moment. We expect all Eclipse projects to be resourceful with common resources. If we ever get close to the concurrency limits, we will discuss it with GitHub.

Hello Frederic,

After much deliberation, the OSEE team has chosen to move to GitHub. We would like to request a dedicated organization “eclipse-osee” in GitHub and a new project repository “org.eclipse.osee”. Please do not make changes to our current Gerrit repository or our Jenkins instance as it will take us time to situate ourselves in GitHub.

Thank you for your time.

Once you are ready to move I'd recommend that we import the Gerrit repos as new repos in the GitHub org. This is usually the easiest way.

While we are setting up and not ready to shut down Gerrit, is mirroring an option?

Unfortunately, mirroring is not an option.

Ah, that is unfortunate. I guess we can do some manual syncing while we get GitHub Actions sorted out.

This hasn't been an issue for other projects in the past and I wouldn't recommend it, but suit yourself.

Good afternoon, Frederic.

I had assumed that project leads could create repositories, but I have been informed that they are unable. Let me know if that is inaccurate.

Under the assumption that is the case, if you perform an import of the Gerrit repositories into GitHub, will that have any impact on the current Gerrit repositories? We would like to have the osee and ote repositories in GitHub, but we wish to keep the Gerrit repos operational for the time being. It is okay if the GitHub repos fall behind the Gerrit ones, we just want to have some source code in GitHub to begin putting pieces together and test.

Thank you for your time.

I had assumed that project leads could create repositories, but I have been informed that they are unable. Let me know if that is inaccurate.

With our self-service, you are able to create a new repo. Here is an example of a PR for such a change: https://github.com/eclipse-4diac/.eclipsefdn/pull/1/files

In case of an import, it's easier and cleaner to do this through the GitHub WebUI though.

Under the assumption that is the case, if you perform an import of the Gerrit repositories into GitHub, will that have any impact on the current Gerrit repositories? We would like to have the osee and ote repositories in GitHub, but we wish to keep the Gerrit repos operational for the time being. It is okay if the GitHub repos fall behind the Gerrit ones, we just want to have some source code in GitHub to begin putting pieces together and test.

Importing a repo from Gerrit to GitHub does not have any impact on Gerrit repos. If you don't want to do a hot switch, I can import the Gerrit repos into "test" repos to make it clear that they are not yet the official source repos.

Hello Frederic - I am chiming in to approve the request as the OSEE product owner - let me know if there is some action required for approval other than this statement of approval for this request.

I concur.

Since the OSEE team has decided to move to GitHub, do I assume correctly that the empty GitLab repos

can be deleted and that https://gitlab.eclipse.org/eclipse/osee/osee-website should be moved to the new GitHub org?

The website can be moved to GitHub and the https://gitlab.eclipse.org/eclipse/osee/org.eclipse.osee can be deleted. Deleting https://gitlab.eclipse.org/eclipse/osee/org.eclipse.ote will require approval from Michael Masterson.

/cc @mmasterso please approve

@fgurr

I approve deleting https://gitlab.eclipse.org/eclipse/osee/org.eclipse.ote. Nothing has been pushed to the source or wiki branches.

Thanks for confirming and the fast turn-around @mmasterso!

https://gitlab.eclipse.org/eclipse/osee/org.eclipse.osee and https://gitlab.eclipse.org/eclipse/osee/org.eclipse.ote are scheduled for deletion on 2024-05-01.

The website repo has been moved to https://github.com/eclipse-osee/osee-website.

Committers will gain access once the sync script runs (~2-4h).

The GitLab website repo https://gitlab.eclipse.org/eclipse/osee/osee-website has been set to read-only.

Please test if the osee-website repo on GitHub works as expected. Once you confirm, I will schedule the deletion of the GitLab repo.

I have pushed a change up to the osee-website repo on GitHub. The targeted page does not seem to have been updated after 40 minutes. Is the update period longer on GitHub?

@mward can you assist here?

Hmm, looks like a logic fault was preventing the tooling from correctly updating. A fix has been made live so this should be fixed.

I can now see my most recent commit reflected on the page. Thanks!

A GitHub bot (ID: eclipse-osee-bot) has been created. Credentials have been added to the OSEE JIPP.

The recommended way to set up a job that builds pull requests is to use a Multibranch Pipeline job (a Jenkinsfile in your repo is required):

New item > Multibranch Pipeline
Branch Sources > Add source > GitHub
Select credentials "GitHub bot (username/token)"
Add the repository URL
Configure behaviors
Save

By default, all branches and PRs will be scanned and dedicated build jobs will be created automatically (if a Jenkinsfile is found).

Has an organization been created for OSEE? I was unable to find it, but I may be searching for the wrong thing or in the wrong place.

Yes, I should have mentioned that: https://github.com/eclipse-osee.

I'm still setting up the self-service. Will report back, when it's done.

Otterdog has been enabled for the eclipse-osee organization.

The following repo has been created: https://github.com/eclipse-osee/.eclipsefdn

You can find the current configuration at https://github.com/eclipse-osee/.eclipsefdn/blob/main/otterdog/eclipse-osee.jsonnet

A dashboard UI of the configuration can also be accessed at https://eclipse-osee.github.io/.eclipsefdn/ which also provides a playground to test snippets before making a PR. When you want to suggest changes, fork the repo, make changes and create a PR. A workflow will automatically run and show you the changes that will be applied and validate that the configuration is correctly formatted and composed.

The documentation of all supported settings can be found here: https://otterdog.readthedocs.io/en/latest/reference/resource-format/

A list of all Eclipse projects that have it also enabled can be accessed here: https://eclipsefdn.github.io/otterdog-configs/ That is often quite helpful to get ideas about how others do their configurations.

If you have any problems, let us know.

What are the rules surrounding pushing to the container registry for our organization?

In the handbook it says that "All members of the committers team are granted the Write role on their project resources hosted on GitHub." However, when attempting to push to our container registry after successfully logging in to ghcr.io, I receive the following error "unexpected status from POST request to https://ghcr.io/v2/eclipse-osee/osee-postgres/blobs/uploads/: 403 Forbidden". Am I to assume that committers do not have write permissions to the container registry?

If that is accurate, how can committer triggered workflows be given permission to push to the container registry? If that is inaccurate, what secret sauce might I be missing?

Am I to assume that committers do not have write permissions to the container registry?

Right.

You have to use the GITHUB_TOKEN secret.

    - name: Log in to ghcr.io container registry
      uses: docker/login-action@v3
      with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

This is a special token created for workflow.

Ah, I see. Thank you! I had been using github.actor rather than github.repository_owner.

I made the modification to use github.repository_owner and still no dice unfortunately. I get "Login Succeeded!" but a "403 Forbidden" when pushing. Are there any additional caveats? For example, does the action need to have been triggered by a user of a particular level? Does the action need to be run on the default branch?

I have never tested myself to push to the github registry. The only caveat I see is if the PR is from a fork. You can take a look at this script from the kuksa project maybe that would help: https://github.com/eclipse-kuksa/kuksa-actions/blob/main/.github/workflows/check_ghcr_push.yml

Thank you for the link! After some further investigation, the issue turned out to be that the default workflow permissions are set to read for packages. Modifying that permission for the workflow solved the issue.

great!

assigned to @heurtemattes

@broberts3cp @grumble What is the status of the migration? anything else is needed?

It has been slow going. It seems there is always something more pressing to be working on, but I believe we are finally picking up steam. The team seems to be on board with the move and I now have several tasks to begin setting up GitLab Actions. I may end up reaching out again if I encounter any roadblocks and I appreciate all your support.

While I have you though, I do have one question. When initially creating the repository I had assumed that otterdog would make an orphaned branch for the specified default. Unfortunately, it created an initial commit on that branch that prevents me from merging any existing branch histories from our other repository. What is the best way to wipe out the current main branch and replace it with an alternate history?

My current game plan is to change the default branch to a temporary branch, delete the old main branch, push up a new main branch with the desired history, then change the default branch to the new main branch. Is their a better way to go about achieving that?

My current game plan is to change the default branch to a temporary branch, delete the old main branch, push up a new main branch with the desired history, then change the default branch to the new main branch. Is their a better way to go about achieving that?

This is a possibility you already create newMain branch, or we can allow temporary force pushes on the main branch, which would allow you to rewrite history.

A force push sounds like the fastest and most straight forward way to get the histories in alignment. Is that something our team can self service or do we need to coordinate it with you? Alternatively, if it is something that needs to be coordinated, it might be easiest for someone with the power on your end to take the dev branch from our Gerrit repo and force push it to the main branch of the GitHub repo rather than trying to time things.

I can get one of our team leads on board if approval is necessary.

I have pushed the master branch from https://git.eclipse.org/c/osee/org.eclipse.osee.git to the new main branch: https://github.com/eclipse-osee/org.eclipse.osee

added team:releng label

I think we're good here. I'm closing the ticket, please reopen if this happens again.

closed

OSEE Migration to gitlab or github

Summary

Designs

Child items ...

Activity