Project repository moves for tools.mat
As part of our ongoing plan[1] to modernize our services we are now moving project repositories from git.eclipse.org to Gitlab/Github.
Webmaster needs to know:
- If the project would prefer to move their code repositories to Gitlab or Github
- What timing would be best for the project. By default it will be the due date of this issue.
- What should happen to the open bugs for the project.
We'll coordinate via this issue and once the move is complete committers will need to update any local copies of the repositories, as well as any build jobs to point at the new remote URL locations.
If we haven't heard from the project by the due date for this issue, Webmaster will request a review from the EMO team to make sure the project is still active. Project repositories for active, but unresponsive projects will be moved to the Foundations Gitlab instance by default, and their bugs will be archived.
/cc @ktsvetkov
Activity
assigned to @mward
Hi Matt,
After a short clarification on the MAT mailing list, I think we can start planning the migration for tools.mat. Here the answers to the questions above:
- We prefer to move to Github
- our website repository moved to github, and this is what the committers prefer also for the code repository
- We would prefer to have the move as soon as the webmasters can do it
- at present we have a rather calm period with very little change. I see no need to wait for August, and we will be happy to move earlier, if this fits into the webmasters schedule
- I'd like to have the open bugs transformed into github issues on the newly created github project (AFAIU this would be possible)
- we have about 50 open bugs. Some of them are open for ages, but I still see value in having them transferred. I will check if some of them could be closed instead of moved.
Is this information enough for you to plan the migration on your side?
Regards, Krum
- We prefer to move to Github
Ok, I've scheduled this move for Wednesday may 8th at 11am Eastern(5pm CET).
Github is fine, and committers should take this time to add their Github IDs to their Eclipse.org accounts(if they haven't already) as that means our tooling can add them once the repo moves.
Importing the open bugs is certainly possible, but just be aware that there is some data loss due to limitations in Githubs API.
Alright, https://github.com/eclipse-mat/org.eclipse.mat has been created and the content from Gerrit has been imported. Committers that have added their Github IDs to their Eclipse.org accounts and enabled 2FA(On Github) should receive access in an hour or so when the sync tool runs.
The Gerrit repository has been marked read-only and a pointer to the new location added to the description.
Bugs are currently being imported(sorry if it's a bit 'noisy'), which is going to take a few go's(due to API rate limiting) and the Bugzilla entry has been updated.
At this time I think this specific move is 'done', but I'll leave it open until the Bugzilla move is done.
That said there are a couple of extra things for the project to consider:
-
Moving the MAT website repo from the Eclipse organisation to the eclipse-mat organisation. We can do that asa part of this, or separately(see #3571).
-
If the project wants to enable self management via our Otterdog solution. In a nutshell Otterdog:
- allows project teams to see the current configuration and suggest changes thus goes into the direction of a self-service process to administrate GitHub organizations (inlucding setting up new repositories)
- allows the Eclipse Security Team to monitor security related settings for our organizations / projects at scale and suggest changes to further improve the overall security of our development processes
Edited by Matt Ward-
- Moving the MAT website repo from the Eclipse organisation to the eclipse-mat organisation. We can do that asa part of this, or separately(see #3571).
Let's not wait with this. I've moved the website repo to https://github.com/eclipse-mat/mat-website and fixed the PMI settings.
-
A GitHub bot (ID: eclipse-mat-bot) has been created. Credentials have been added to the MAT JIPP.
The recommended way to set up a job that builds pull requests is to use a Multibranch Pipeline job (a Jenkinsfile in your repo is required):
- New item > Multibranch Pipeline
- Branch Sources > Add source > GitHub
- Select credentials "GitHub bot (username/token)"
- Add the repository URL
- Configure behaviors
- Save
By default, all branches and PRs will be scanned and dedicated build jobs will be created automatically (if a Jenkinsfile is found).
A webhook has been created that will trigger builds on pushes and pull requests once a Multibranch Pipeline job has been configured.
@fgurr I can't fully understand what is available and what I still have to do. Sorry, never done this before... so I started looking for a more detailed explanation. I found https://wiki.eclipse.org/index.php?title=Jenkins&redirect=no#GitHub_integration, but sill struggling... therefore I'll ask here.
So far we didn't trigger builds for gerrit contributions. I.e. I might look at this later, as a new improvement, that builds are triggered also for PRs, but for now I'd like to get as quickly as possible to our previous state - polling for changes once an hour :) Is this still possible, or is it mandatory to use a webhook?
About the webhook - do I understand correctly - this is already setup by you, and we don't have to configure it explicitly? If so, what should I configure in jenkins (or is my only option to use the Multibranch Pipeline mentioned above)?
I guess it might be easier for me if I see an example. Could you point me to a project which I could use as a sample or give me some more details what I need to do.
-
Is this still possible, or is it mandatory to use a webhook?
The webhook does not block you from still using polling. Polling is discouraged since it stresses servers unnecessarily.
About the webhook - do I understand correctly - this is already setup by you, and we don't have to configure it explicitly? If so, what should I configure in jenkins (or is my only option to use the Multibranch Pipeline mentioned above)?
Yes, the webhook is set up already and if you are using a Multibranch pipeline job then no additional setup is required.
I have created an example multibranch pipeline job here: https://ci.eclipse.org/mat/job/ef_releng_test_multibranch/
It points to a personal fork of https://github.com/eclipse-mat/mat where I added a Jenkinsfile: https://github.com/fredg02/mat/blob/master/Jenkinsfile
The multibranch pipeline job can be configured to ignore PRs until you are ready to use them.
If you agree with this approach, I can create a pull request for the Jenkinsfile and you can adjust it as you like.
-
Does it make sense to use GitHub Actions for CI?
It depends on your use-cases. Some Eclipse projects use either Jenkins or GitHub actions, or even both.
I guess it would be easier to understand for contributors,
Depends on your contributors. Both CI systems have their pros and cons.
and it would be more secure. For instance, running untrusted code from PR in Eclipse Jenkins is not very secure.
Running untrusted code is not very secure in any CI system.
@fgurr Thanks a lot for your help! Having a build for the PRs is definitely a good improvement, so I'll take some time and look at the pointers you provided. I just wanted to get back to the "normal" state.
@vsitnikov I am really happy with the service we get with JIPP. We don't have to care about anything than our own job definitions, which we touch very rarely - everything else is part of the service. And we are used to it, which is a huge benefit with the limited time the few committers can spend on MAT. At present I see no need to look for a different solution (for MAT).
Thanks for moving the repositories! I see we never talked about naming, and I made some (wrong) assumptions. I was not aware of #3571 and was hoping to get /eclipse/mat (as we had already /eclipse/mat-website). I understand the change is going to be for all projects, so I guess we'll have to live with it.
In case the decision is final and all projects will move to their own organisations - could we at least rename the code repo to "mat", i.e. that we have at the end /eclipse-mat/mat and /eclipse-mat/mat-website? If we have an own organisation, there is no need to have the namespace in the project name too.
And in case we still had a choice, I would really prefer staying under the eclipse organisation (/eclipse/mat), as we are not expecting any additional repos coming (the project has been around for 16 years with a single code repository + the web site).
-
could we at least rename the code repo to "mat"
Done.
And in case we still had a choice, I would really prefer staying under the eclipse organisation (/eclipse/mat), as we are not expecting any additional repos coming (the project has been around for 16 years with a single code repository + the web site).
That's not an option, sorry. We need to split the eclipse org up into individual GH orgs for security reasons.
I understand it might be a bit late, however, it might be better to use bulk import GitHub API so the imported issues have the proper historical timestamps, they do not trigger notifications during import (so the they do not overload GH servers).
Previously, I migrated Apache JMeter issues from Bugzilla to GitHub (including attachments) with a help of https://github.com/vlsi/bugzilla2github, see https://github.com/apache/jmeter/issues
The best docs that I know of are at https://gist.github.com/jonmagic/5282384165e0f86ef105
While I can see your point, my experience is that old(and closed) bugs are generally less interesting unless someone is facing a specific issue with an older version, and since most people start with Google rather than Github, they'll still find this content on our Bugzilla instance which is fairly well indexed.
For active members of the MAT community, they know where to find the old bugs, and newcomers are more likely to be interested in the open issues where they can actually contribute.
So the question isn't "Can it be done?", but rather "Should it be done?", and I don't think so.
Done.
Thanks, Frederic!
- If the project wants to enable self management via our Otterdog solution.
I will look at Otterdog and come back to you.
I've moved the website repo to https://github.com/eclipse-mat/mat-website and fixed the PMI settings.
I checked the project page and under developer resources (https://projects.eclipse.org/projects/tools.mat/developer) I see now towards the bottom of the page that the project is at Github and a link to the organisation (not to the code repo?). The focus however is still on Gerrit and on outdated links to gerrit / old git repo (see attached image below). I didn't find a way to remove these myself. Could you help me how to do this, or is it something only the you can maintain?
@ktsvetkov I have removed the Gerrit configuration.
@ktsvetkov I have removed the Gerrit configuration.
@heurtemattes thank you!
If the project wants to enable self management via our Otterdog solution.
@mward I don't understand well what we are supposed to do. Let me try to describe what I understood, then you can tell me how far it is from the truth...
- if we are interested in self-management via Otterdog, then somewhere there will be a separate repository (which you create??) with the configuration for the eclipse-mat organisation
- with this, we'll be able to see what the settings for our organisation and repositories are
- if we need changes (e.g. new repository under), then we'd need to open a PR on the repo containing the configurations
If my understanding is right, then we'd just need to say "yes" for now - is this correct?
Yup, that's pretty much my understanding.
You can checkout https://github.com/eclipse-platform/.eclipsefdn/tree/main/otterdog if you'd like an example from another project that is already using Otterdog .
I think we could finalise the move, which went pretty smooth. Thanks! To summarize some of the open discussions:
- committers access
- one of the project committers (jasonk000) wasn't able to close issues on the project and we figured out he is not a member of the eclipse-mat organisation and he hasn't received an invitation. We supposed this might be because of missing link between eclipse Id and github ID, but Jason has yet to confirm. If this was the case, will he automatically be added later (when the ID is maintained) or shall we open a ticket for it?
- I see that ajohnson1 is also not in the organisation, but there I can't tell if we received an invitation and has not yet accepted, or if there is a different reason. He should have the ID maintained properly, as he had already permissions for the website repo, which was moved some months ago.
- otterdog - thanks @mward for the example. I think it would be useful to have if for MAT. Is there something I should do, or are the webmasters going to create the additional repo and the content in it?
- jenkins build - with the available hooks I was able to switch from "Poll SCM" to "GitHub hook trigger for GITScm polling" for our "master" branch build and I think this is good enough for now. Thanks @fgurr for the examples of a multibranch build - I think I understand how we could use the feature. However, I am still concerned, that with an automated PR build we would give anyone the ability to run some code on our Jenkins, without any review, and I am still considering if we should that or rather look for an easy way for committers to trigger a branch build after they looked at the changes. Anyway, thanks again for the support and I think the rest we should be able to handle on our own.
- committers access
-
committers access
I confirm Jason hasn't provided his github handle in the eclipse profile. A script is running every 2h, so he will be added automatically after updating this profile. No need for a ticket. An invitation has been sent to Andrew Johnson and it is still pending. Can he check his mailbox?
Otterdog
I will initiate the setup.
jenkins build
The Multi-branch pipeline plugin allows fine-grain configuration on PR build strategies and policies. You can define
Discover branchesstrategy,Discover pull requests from originStrategy,Discover pull requests from forksStrategy. Default configuration of a new MultiBranch Pipeline project in jenkins has a sufficient level of security for PR. This is what we recommend for projects. -
otterdog has been enabled for the eclipse-mat organization. The following repo has been created: https://github.com/eclipse-mat/.eclipsefdn
You can find the current configuration at https://github.com/eclipse-mat/.eclipsefdn/blob/main/otterdog/eclipse-mat.jsonnet A dashboard ui of the configuration can also be accessed at https://eclipse-mat.github.io/.eclipsefdn/ which also provides a playground to test snippets before making a PR. When you want to suggest changes, fork the repo, make changes and create a PR. A workflow will automatically run and show you the changes that will be applied and validate that the configuration is correctly formatted and composed.
The documentation of all supported settings can be found here: https://otterdog.readthedocs.io/en/latest/reference/resource-format/
A list of all Eclipse projects that have it also enabled can be accessed here: https://eclipsefdn.github.io/otterdog-configs/ That is often quite helpful to get ideas about others do their configurations.
Otterdog is also deployed as github app and its commands can be invoked from within PR: https://github.com/eclipse-csi/.eclipsefdn/pull/6
If you have any problems let us know.
assigned to @heurtemattes
added team:releng label
I think we are done with the move for MAT. Feel free to close this ticket. And before you do, I would misuse it for an additional question one more time :) With bugzilla there was some integration in the project pages (https://projects.eclipse.org/projects/tools.mat) - such, that when we marked bugs for a specific milestone, they automatically appeared in the corresponding release record. Is there a similar mechanism for github repos (i.e. marking issues with a specific label or alike that would result in the issues being shown in a release record)?
