Consider supporting arbitrary nesting of groups

For consideration...

As I understand it, GitLab supports arbitrarily nested groups.

Our standard practice is allocate resources "flat", independent of project structure. My observation is that with GitLub, "flat" means that we're creating a group for each project under the "eclipse" group, e.g., "/eclipse/dash", and all repositories associated with the corresponding project are created in that group.

We adopted the "flat" notion quite some time ago to mitigate the amount of work and negative effects of having to move resources following project restructuring (e.g., when projects move from one TLP to another).

That GitLab provides support for arbitrary nesting of groups, sets up automatic redirects for moved repositories, and otherwise provides what I assume are reasonably good management tools, presents an opportunity to reconsider the policy.

I'm thinking that a default position is to have a distinct group for each project to mitigate the risk of confusion regarding which project owns which repository (and by extension which developer has commit rights on which repository), but allow that project groups can themselves be arbitrarily organized into groups.

From the @webmaster workload perspective, I believe that we'll have to manage expectations that the webmaster will not be able to continuously tweak and move groups around.

Would allowing arbitrary nesting introduce significant work in the provisioning process? Are there scenarios that are particularly onerous?

For one, at very least, keeping track of arbitrary nesting and what goes where will require some additional thought/management/tracking.

@cguindon are permissions assigned at the group level or the repository level? Does the script that manages committer and project leads care about the group nesting?

@cguindon are permissions assigned at the group level or the repository level? Does the script that manages committer and project leads care about the group nesting?

At the group level. For example, Eclipse Titan: https://gitlab.eclipse.org/eclipse/titan

Committers were added to the https://gitlab.eclipse.org/eclipse/titan project which grants them access to all the sub-projects within that space!

The sync script is responsible for creating this nested structure.

So that confirms the need to have a specific group for each project. I like that requirement to help keep the relationship between projects and potential multiple repositories clear.

Do the scripts care about nesting? e.g., if we were to create a repository, /eclipse/stuff/dash/stuff.git, would the script create permission groups for the "/eclipse/stuff/dash/" group?

Do permissions trickle up/down to the "/eclipse/stuff" group?

@wbeaton can you give an example of how "project groups" could be re-arranged? IIUC, you are referring to a structure similar to the top-level projects and their (sub-)projects, e.g.

• group "tools"
  • subgroup "acute"
  • subgroup "ajdt"
  • subgroup "cdt"
    • repository "cdt"
    • repository "cdt-vscode"
    • [...]
• group "technology"
  • subgroup "cbi"
  • [...]

I anticipate that there may be some top level projects that might want this.

The immediate example I have are two Technology subprojects, AsciiDoc Language and Eclipse Austin that work on different aspects of the same project.

I should note that I started this discussion with an expectation that nesting groups has some value greater than simple organization, e.g., combined analytics, issues, boards, etc.

The main issue I have is that the current situation is now inconsistent. For asciidoc-lang, the group is asciidoc. That gave me the impression we were putting projects under a group for the TLP. Therefore, I expected it to be /asciidoc/asciidoc-lang and /asciidoc/austen (pending approval of the AsciiDoc TLP). If, on the other hand, we are making one group per project (which seems excessive, but if it's a limitation of the tech, okay), then it should have been /asciidoc-lang/asciidoc-lang.

I really don't see any reason why projects can't be created under a group for the TLP*. I think too much concern is being given to fine-grained permissions. To a certain degree, there needs to be some trust (within the TLP) or else you end up over-engineering the solution.

* Obviously, the "technology" TLP would need to be some sort of exception.

Do the scripts care about nesting? e.g., if we were to create a repository, /eclipse/stuff/dash/stuff.git, would the script create permission groups for the "/eclipse/stuff/dash/" group?

By design, The PMI won't allow an admin/webmaster to add a repo outside of https://gitlab.eclipse.org/eclipse/:shortProjectId/[:repo-name]

The sync script takes actions based on data from the API on projects.eclipse.org.

We will need to make some changes to the sync script and PMI if we want to change this convention.

mentioned in issue #43

The immediate example I have are two Technology subprojects, AsciiDoc Language and Eclipse Austin that work on different aspects of the same project.

While I can see certain (management) benefits from the project's perspective, I think arbitrary grouping/nesting can lead to a lot of confusion. Especially if it's different from the structure in PMI. E.g. if a user tries to find the Eclipse Austin source code in GitLab, but can't find it (without using a search box) since it's hidden in a subgroup, I think the price is too high.

Do permissions trickle up/down to the "/eclipse/stuff" group?

"A user’s permissions in a subgroup cannot be lower than in any of its ancestor groups. Therefore, you cannot reduce a user’s permissions in a subgroup with respect to its ancestor groups."

From https://docs.gitlab.com/ee/user/group/subgroups/index.html

So permissions should only be given on the leaf-node projects and the sync script needs to be aware of parent sub-groups.

E.g. if a user tries to find the Eclipse Austin source code in GitLab, but can't find it (without using a search box) since it's hidden in a subgroup, I think the price is too high.

At least in this case, the intention is to provide a grouping to make it easier to build a brand and shared presence around the technology and for the community to find related projects.

At least in this case, the intention is to provide a grouping to make it easier to build a brand and shared presence around the technology and for the community to find related projects.

To Grouping! The Cause Of, And Solution To, All Of Life's Problems https://www.youtube.com/watch?v=SXyrYMxa-VI

Although all projects appear nested under the Eclipse top-level group, we're still maintaining the 1-group-per-Eclipse project. I'm not sure if we want nested groups unless it's something we can automate/self-serve from the get-go.

mentioned in issue #47 (closed)

If we change the nesting of our project on GitLab, we need to change it for all projects or we need to start tracking these nesting relations in the PMI.

The sync script will need to know which TLP should be nesting projects.

Right now the nesting is /eclipse/project/repo.git. Permissions are granted on the group level, so our script configures /eclipse/project with the permissions for the committers.

Is it possible to keep putting permissions on the leaf directory, and just skip over the extra levels? That is, if we add an extra layer of nesting, say /eclipse/tlp/project/repo.git, can we just set the committer permissions on /eclipse/tlp/project? And, in the absence of additional information, just leave with either an empty group or some default? (would that actually work?).

If we then create a repository for the TLP a /eclipse/tlp/repo.git, could we then create a group with the committers for at /eclipse/tlp?

Does GitLab have a notion of a hierarchy of groups? e.g., is there some "risk" that committers would have permissions on ?

Right now the nesting is /eclipse/project/repo.git. Permissions are granted on the group level, so our script configures /eclipse/project with the permissions for the committers.

Correct.

Is it possible to keep putting permissions on the leaf directory, and just skip over the extra levels? That is, if we add an extra layer of nesting, say /eclipse/tlp/project/repo.git, can we just set the committer permissions on /eclipse/tlp/project? And, in the absence of additional information, just leave with either an empty group or some default? (would that actually work?).

We need a group per project to keep things efficient.

We don't want to manage users at the project level since that will require us to add/remove the same committers on every repo/project. The sync script currently takes 1 hour to run and we only have a few projects on Gitlab.

If we then create a repository for the TLP a /eclipse/tlp/repo.git, could we then create a group with the committers for at /eclipse/tlp?

We can't create a group within a group. In your example, the group is TPL, and users added to that group would inherit the same role for any sub-group and sub-project created under the TPL group.

Something to note: It's possible to add existing groups to another group or project.

Does GitLab have a notion of a hierarchy of groups? e.g., is there some "risk" that committers would have permissions on ?

Any users added to the TLP project would retain the same role for any sub-project or sub-group.

• /eclipse/asciidoc - Webmaster (Owner)
• /eclipse/asciidoc/asciidoc-lang - Webmaster inherit the role of owner because of eclipse/asciidoc. It's not possible to remove it without removing webmaster from /eclipse/asciidoc

To conclude, I have a good idea how we could implement TLP nesting. However, I am not convinced yet that this is worth fixing. If we do, it will probably be an exception built-in in our sync script that will always require staff to get involved to make this happen.

I don't want our sync script to start moving repos around. This could break the builds job of a Project since the clone URL will be changed by moving the project to another location.

I think each TLP will have different requirements. For example, if we decide to enable this to the Technology TLP, we will need to pause the sync script, enable TLP nesting in our sync script for Technology TLP, migrate any existing project to the new nesting structure. Once everything is in place, we would need to restart the sync script service.

The process should be simpler if we decide to enable this to a TLP that doesn't have any projects on Eclipse GitLab yet.

mentioned in issue eclipsefdn/emo-team/emo#65 (closed)

mentioned in issue eclipsefdn/emo-team/emo#95 (closed)

@cguindon @wbeaton We'd be very happy @ Eclipse Oniro Working Group to be the guinea pigs for trying out nesting of groups. The flat recommended structure is honestly very limiting to us. We have a fairly nice structure over at https://booting.oniroproject.org/distro with all components nicely separated in the Components sub-group.

Trying to flatten this isn't helping developers in understanding the code structure either. And a project having multiple git repos with different maintainers is more natural in an OS project.

Can we try this out?

cc: @agherzan

@idlethread,

The Asciidoc WG is also requesting this feature. I would like to take some time to better understand your expectations before I commit to making any changes to our structure.

I am looking at https://booting.oniroproject.org/distro, is it safe for me to assume that each group in that list will map to a new Eclipse Project?

I am basically trying to visualize what you are trying to achieve with Eclipse projects in mind.

From what I can see in the PMI, we currently have 2 projects for Oniro:

• TLP Eclipse Oniro: https://projects.eclipse.org/projects/oniro
• Eclipse Oniro Core Platform: https://projects.eclipse.org/projects/oniro.oniro-core

To clarify my question, will we have an Eclipse Blueprint project or are you looking to create a similar structure from https://booting.oniroproject.org/distro under the Eclipse Oniro Core Platform project?

@cguindon

I am looking at https://booting.oniroproject.org/distro, is it safe for me to assume that each group in that list will map to a new Eclipse Project?

Not all the groups. All the meta- prefixed ones will end up under oe below. We're thinking something like this:

oniro-core
 |
 |
 |---- oniro
 |---- docs
 |---- infrastructure/*
 |      |
 |      |
 |      .
 |      .
 |---- oe/*
 |      |
 |      |---- 3rdparty/*
 |      .
 |      .
 |---- components/*
 |      |
 |      |---- 3rdparty/*
 |      .
 |      .

To clarify my question, will we have an Eclipse Blueprint project or are you looking to create a similar structure from https://booting.oniroproject.org/distro under the Eclipse Oniro Core Platform project?

Yes, there will be a separate Eclipse Oniro Blueprint project because the sources will be maintained independently of Oniro Core.

But oniro-core allowing sub-groups and the above structure will make lives much easier for the following reasons:

1. Easier understanding of the project layout when browsing through the web interface
2. We have tens of 3rd party components that we need to integrate into the core OS. They don't belong in a separate project. Having them all at one root level will make things hard.
3. Prefixes don't work for us because certain prefixes have special meanings in Yocto tooling e.g. meta-. Builds will break if we change that.
4. Adding subgroups will simplify this for CQ and ongoing IP reviews since forks can be stored together e.g. in 3rdparty/* above.
5. For now, we can start with a single root-level permissions (committers) for the entire project.
6. But eventually, we'd like the ability to have sub-group and repo-level permissions for committers. We don't want to initiate new committers with committer access to the entire high-level group.

mentioned in issue eclipsefdn/emo-team/emo#168 (closed)

But eventually, we'd like the ability to have sub-group and repo-level permissions for committers. We don't want to initiate new committers with committer access to the entire high-level group.

This is not supported by the Eclipse Foundation Development Process (EDP). According to the EDP, all committers have uniform access to all project resources. If something different is required, then an update to the EDP is required.

mentioned in issue #1034 (closed)

added MBO team:webdev labels

changed due date to June 30, 2022

assigned to @malowe

We are planing to work on this issue in Q2. I've started to work on a document that will describe how we plan to solve this issue. I will share our plans once I am done.

At a high level, we plan on supporting the following features:

1. ability to allow top-level project nesting

For example, a TLP such as Eclipse IoT will be allowed to move their "sub-projects" under the Eclipse IoT TLP Group.

This would translate to:

 | Eclipse (GL Group)
 |-- Eclipse IoT (GL Group)
 |---- Eclipse 4diac™ (GL Group)
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)
 |---- Eclipse Agail (Group)
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)
 |---- Eclipse Amlen (Group) 
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)

2. allow projects to create their own hierarchy

We will allow projects to create subgroups and sub-projects within their Project GL Group. For example, the following structure would be allowed:

 | Eclipse (GL Group)
 |-- Eclipse IoT (GL Group)
 |---- Eclipse 4diac™ (GL Group)
 |------ Examples (GL Group)
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)
 |------ IDE (GL Group)
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)
 |-- Eclipse Web Tools Platform Project(Group)
 |---- Examples (GL Group)
 |------ repo 1 (GL project)
 |------ repo 2 (GL project)
 |---- IDE (GL Group)
 |------ repo 1 (GL project)
 |------ repo 2 (GL project)

However, we won't allow project nesting. It's not going to be possible for Eclipse Paho to be included as a subgroup of the Eclipse Kura project. This is just an example, they probably wouldn't want that in the first place.

I've published additional details about our plans on &12 (closed)

added to epic &12 (closed)

mentioned in epic &12 (closed)

@cguindon Will this proposal handle per-subproject permissions and the necessary changes to EDP mentioned by Wayne?

For broad projects such as Oniro that integrate hundreds of upstream (3rd party) and 1st party code, it helps to have finer grained permissions. Other Linux distributions (e.g. Debian, Ubuntu, Fedora) start community members off with package-level maintainer permissions and gradually allow them to become core maintainers.

I believe you are asking if we are going to allow projects to grant developer roles to non-committers? If that is the question, then no.

However, if you are referring to forks (3rd party repo), then we might be able to make an exception since we already plan to disable ECA validations on those types of projects in order to allow projects to pull in changes from upstream which would include commits from people without an ECA.

I would need @wbeaton to make the request since it's not something we've done in the past and I want to make sure he's ok with any changes in our contribution workflows.

Will this proposal handle per-subproject permissions and the necessary changes to EDP mentioned by Wayne?

An EDP change would be required first. I need a very compelling argument to start that work, since it will be very significant change and a represents significant investment of time. By way of expectation management, changes to the EDP generally take months to author and socialise and must be approved by the board of directors.

@cguindon I'm asking for subgroups to allow different roles to developers instead of having a single set of roles at the TLP. Gitlab would then allow the developer the maximum role on a repo at the leaf of a the nested subgroup hierarchy by traversing the tree.

e.g. Considering your example above:

 | Eclipse (GL Group)
 |-- Eclipse IoT (GL Group)                          <--- devA:Committer, devB:Committer
 |---- Eclipse 4diac™ (GL Group)                     <--- devC:Committer
 |------ Examples (GL Group)                         <--- devD, devE:Committer
 |-------- repo 1 (GL project)
 |-------- repo 2 (GL project)
 |------ IDE (GL Group)                              <--- devD:Committer
 |-------- repo 1 (GL project)                       <--- devE:Committer
 |-------- repo 2 (GL project)                       <--- devF:Committer
 |-- Eclipse Web Tools Platform Project(Group)
 |---- Examples (GL Group)
 |------ repo 1 (GL project)
 |------ repo 2 (GL project)
 |---- IDE (GL Group)
 |------ repo 1 (GL project)
 |------ repo 2 (GL project)

This would allow for developer F (devF) who has done good work on IDE repo 2 to be made committer along with devD. This allows community contributors to be "promoted" to committer to small areas of the project based on their contributions. As they gain more experience and trust of the other committers they may be promoted to committer on larger parts of the project.

For Oniro, this is critical to allow us to grow a community of contributors w/o necessarily allowing them to commit to all areas of the project. We have too many repos (3rd and 1st party) that a few TLP-level committers can not manage all by themselves.

Without this, group nesting is only partially useful - It improves code layout but doesn't allow for sharing of committer responsibility.

@wbeaton I understand it'll take time to socialize this change to EDP. How can we help get the ball rolling on that discussion?

@wbeaton I understand it'll take time to socialize this change to EDP. How can we help get the ball rolling on that discussion?

I'm not trying to be difficult, but, frankly, this sort of change will have all sorts of implications and I do not want to engage in this lightly. "I need a very compelling argument to start that work" and I haven't gotten that yet. By way of background, many years ago, we did have a notion of components within a project, each managed with different ACLs managing who could access what and it was a mess. So I need a very compelling argument if we're going down that path again. Further, if there are to be multiple classes of committers, then we'll have to have some understanding of how membership in the various classes is managed in an open, transparent, and meritocratic manner: we'll need to codify the means by which somebody gets included in another group, and that means can't be "so-and-so says so".

In general, the way that we change the EDP is for the corresponding PMC's representative on the Eclipse Architecture Council to open an issue and champion the change. So, our first step is to identify the Architecture Council representative for Oniro. I believe that this has been discussed with the PMC but I'm happy to join a call to discuss if that's required.

In the meantime, our existing model has worked well for a variety of scenarios at least a decade. Contributors can interact with the project on any and every repository via merge requests. If a contributor cannot be trusted to stay out of areas where they do not understand what they're doing, then they shouldn't be elevated to the position of committer. We have several very large projects that successful manage who should access what by this sort of "social convention" today.

If some aspects of the project must be more exclusive than others, then perhaps the project should be factored into two or more separate projects, with a higher bar for participation on the project with the more sensitive bits.

I am looking at updating the documentation to this under the README for this and I have a question. I'll be looking at migrating this into the handbook eventually, so I want to be sure that this is accurate. I'm looking at updating the section around the options for project namespaces, and I don't think we've defined who can request updates of the namespace group display names. An example of what I mean by display name is for eclipse/titan, the display name is Eclipse Titan.

Would the people who request this update be the project leads, or would there be another group that would request changes to display name?

1. Using the short project ID as the path of their group (i.e asciidoc-lang, dash). While the path of the project is determined programmatically, a display name may be chosen by the _____ of the project and a request made through the Eclipse Foundation Gitlab Helpdesk or requested at the time of the creation of the group.

2. Nesting the group mentioned in the first option within a group for the TLP (top-level project) of the given project. (i.e. asciidoc/asciidoc-lang, technology/dash). The name of the TLP group is determined by ___, and similarly to the first option, a request may be made via the Eclipse Foundation Gitlab Helpdesk.

The root group for a project should use the project's formal name (e.g., "Eclipse Titan" rather than "Titan") as the display name. Anything other than minor variations of that should be considered exceptional.

For subgroups, my initial thought is that these can be named arbitrarily with approval from the project lead (any committer could ask, but we'd have to verify with a project lead).

While the path of the project is determined programmatically

We do periodically restructure projects. Recently, for example, we moved a bunch of projects from the Eclipse Technology TLP to the new Eclipse Automotive TLP. During these moves, project ids changed (e.g., technology.sumo became automotive.sumo). Changing a project's id should not necessitate a change in the repository structure (the project team and/or PMC may separately request it).

added statewip label

@wbeaton we were testing this earlier today and @cguindon and I realized that there is a secondary effect to some of these group nesting actions. We can have elected members of TLPs gaining access to subprojects outside of election process. I'll use asciidoc as an example of it.

For the Asciidoc TL project, we can set the group as eclipse/asciidoc (as that is the project name for the TLP). Currently, we can also set the Gitlab namespace to eclipse/asciidoc/asciidoc as technically the project is part of the asciidoc TLP space.
For the Asciidoc Language project we can have the either the current state of eclipse/asciidoc-lang or the new nested group of eclipse/asciidoc/asciidoc-lang.

If we set the Gitlab namespace as the nested example for the asciidoc-lang project, then because of the ways that permissions are inherited anyone with rights in the TLP will have those same rights for subproject repos in Gitlab. This has side effects that could be either positive or negative depending on outlook, so we wanted your input on it, and whether we should allow/block this. Do you have any preferences?

At least some (I want to say most, but that may not be supported by the data) of our top level projects don't have committers. In cases where the TLP does have committers, those committers should also PMC members (that should generally be true) who at least theoretically have a better understanding of our rules than the average committer. It's also not entirely clear to me that a PMC member authoring a commit here-and-there for a subproject under their purview is necessarily a violation of the EDP. Further, there's considerable overlap between TLP committers and subproject committers, so I believe that the actual problem space is relatively small.

In any case, the committers in a TLP would have signed the same committer agreements as everybody else, so in the general case, there's no IP exposure.

It's less than optimal for unelected developers to have committer access to project repositories, but my concern for this is greatly overshadowed by the IP exposure concern (which I believe that I've asserted doesn't exist).

I'm inclined, therefore, to recommend that we just document this issue and move on.

Sure, I can add this to docs just so its understood its a side effect :)

Other than updating documents, this is live. I will start updating some documents today re what Wayne and I spoke of above.

We now have everything in place to support 1) top-level project nesting and/or 2) allow projects to create their own hierarchy by creating sub-groups.

We plan to test the migration tomorrow with Eclipse Dash. This will allow us to catch any major issue. Once we confirm that all is working well, we will be ready to roll this out to any project who makes the request.

mentioned in issue #1525 (closed)

marked this issue as related to eclipsefdn/it/api/git-eca-rest-api#71 (moved)

removed the relation with eclipsefdn/it/api/git-eca-rest-api#71 (moved)

As an update, this has been rolled out to Dash, and we are going to be testing this functionality with Oniro to iron out deployment wrinkles and write real-world migration documents/steps. We have a deadline of finishing that migration in the next month or so (iirc), and then we will make an announcement of the new feature to the general public.

@malowe What is the current status of the migration? Are we ready to make a public announcement?

https://gitlab.eclipse.org/eclipse/technology/dash Dash is stable, no issues reported. Oniros issue for migrating their excluded groups is still open #1532 (closed)

mentioned in issue #1703 (closed)

This has been live and stable for 6+ months now, though it is underused. Do we want to close this, or make a concerted effort to get more samples to test? Our previous CTA has no engagement, so we were unable to confirm it.

@malowe I am thinking before we can close this issue we should make sure this is properly documented in the Project Handbook.

Can you take care of that, please?

eclipse/technology/dash/org.eclipse.dash.handbook!11 (diffs)

changed due date to July 31, 2023

changed due date to June 30, 2023

The handbook has been updated and is now live with the updates on the ability to use this feature https://www.eclipse.org/projects/handbook/#resources-gitlab. With that, we will mark this as complete

closed

mentioned in issue #5073 (closed)

marked this issue as related to #4764