Rethink how we represent simultaneous release participation

For every release, we (I) compile a participating projects page (e.g. 2022-12) and initiate a dance over cross-project-dev-issues to sort out specific versions included in releases and whatnot. These pages are generated using records that map a specific simultaneous release instance to specific release records for individual Eclipse open source projects.

The association between the simultaneous release and project release record is used to automatically generate participation statements in the PMI.

e.g., for a release record you'll see something like this:

"This release is part of ..." is generated based on the association. A similar statement is generated on the project's PMI page as well.

We also use this information to generate stats that we include on the various simultaneous release marketing pages. For the purposes of running states, we only need the ID of each of the participating projects (that is, we don't actually use the release version information). From that ID, we can identify the repositories owned by the various participating projects and run queries to collect stats (contributors, committers, lines of code, ...) based on that.

There's a couple of corner cases where we do actually leverage the release records to get information about participating subprojects. For example, the Eclipse project participates in the Simultaneous Release as a single project, but actually includes content from some -- but not all -- subprojects (the list of subprojects is included in the release record). I believe that it's only the Eclipse and and Eclipse Web Tools projects that do this.

I'd like to rethink how we do this. The disconnect between the work that committers do and the tracking that we do for the release results in a lot of very manual work to keep the information in sync, which introduces potential for errors. That, and it's not clear that the manual work actually adds value.

As an aside: It may be time to just move on from release records and just let projects do releases however they feel is best for their community. If we need release information, we can query it from GitLab and GitHub. Whatever we decide here could be a step towards making that happen.

I've made some passes at making reasonable guesses at participating projects from the aggrcon files, but there's enough variety in them that it would require a lot of hinting. We can correctly guess, for example, that the fact that cdt.aggrcon exists in the repository and is enabled means that tools.cdt is participating in the release. But there are some aggrcon files that represent multiple projects and cases where single projects have multiple aggrcon files. I tinkered a bit with making good guesses based on the repository records in the document, but even that requires some educated guesses.

And, frankly, I'd rather not guess. Further, I'd like to see what we can do to completely automate all the things that we do with this data, and requirements manual intervention would impact that.

I'm thinking that it would be useful to include project IDs directly in the aggrcon files. The primary idea being that having the information all in one place improves our chances that it will be maintained and therefore correct. If we decide that version information is no longer interesting, it may be the case that we can just update these files with the information once and then never have to think about it again. If version information is considered interesting, then we'll need to capture that here as well; and that that may require periodic "thinking about".

We'd need to be able to list multiple project IDs. For example, ep.aggrcon would need to include eclipse.platform, eclipse.equinox, eclipse.jdt, ...

Questions...

Is the release version information useful to anybody?

Is including the "This release is part of..." statement on release and project records in the PMI valuable?

Based on responses to the above, here's what I'm thinking:

• we make it a requirement to put project IDs into the aggrcon files
• we remove the "This release is part of ..." functionality from the PMI
• we snapshot data extracted from the aggrcon files and use it to generate the simultaneous release participating project pages
• we remove version information from the simultaneous release participating projects pages

This is just a brain dump. I'm open to other ideas.

/cc @emerks @jograham

assigned to @wbeaton

marked this issue as related to #138 (closed)

Note that the "same" information is already captured in this "decorator" model:

https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/tree/simrel.aggran

This model reconciles the information using PMI rest APIs. It has a link to the corresponding project, i.e., site="..." for each contribution. It knows the release date of SimRel and can paw through the release records to find highest recorded on or before the SimRel release date. It can also find all the git repositories associated with each PMI project. In the Analyzer Editor analyzes those for activity as well as the history of commits on the *.aggrcon of each contribution.

The goal of this is to eliminate the manual processing. I've got some hacked code in my local version to compare what you send out to what's generated in this model, where mostly there are only a few small differences that in the end, I think no one will notice or care about...

I think the simrel.aggran file is a good way to manage this going forward. Adding the extra fields needed to this file to achieve the requirements is best.

As an aside: It may be time to just move on from release records and just let projects do releases however they feel is best for their community. If we need release information, we can query it from GitLab and GitHub. Whatever we decide here could be a step towards making that happen.

+1

One of the biggest benefits of the collating the release records is making the overall N&N that now lives on https://eclipseide.org/release/noteworthy/. That said, as more and more projects are using single pages for N&N that list becomes more static as time goes by and I think that is a good idea.

marked this issue as related to #146

FYI, here is a comparison of the differences between what is generated for simrel.aggran versus what's here:

• https://projects.eclipse.org/releases/2023-03

These are projects whose contents are contributed to the train by someone else or are essential for producing the trains contents:

• Eclipse e(fx)clipse=3.9.0
• Eclipse Jetty - Servlet Engine and Http Server=11.0
• Eclipse Nebula - Supplemental Widgets for SWT=3.0.0
• Eclipse Orbit Project=null
• Eclipse Tycho=3.0.3

So yes, not really direct participating maintaining a *.aggrcon file, but without these, there would be missing content or we'd not be able to build the content at all. These projects are considered "part of the IDE WG's set of projects".

These projects have different versions:

• Eclipse Sirius sirius-web-2023.3.0 != 7.0.7
  • They contribute https://download.eclipse.org/sirius/updates/releases/7.0.7/2021-06/
• Eclipse Wild Web Developer 1.0.3 != 1.0.2
  • https://download.eclipse.org/wildwebdeveloper/releases/1.0.2/
• Eclipse WindowBuilder 1.12.0 != 1.11.0
  • https://download.eclipse.org/windowbuilder/1.11.0

So these differences are all because the project has a release scheduled before the SimRel release but one that they did not contribute to SimRel. A deeper analysis of the versions of contributed content, assuming there is content with a "representative" release version number, could help address such issues. Or one might look at version numbers in the p2 site URL, assume that pattern continues to be followed...

This one should be removed because XSD has been rolled up into EMF:

• Eclipse MDT XSD (XML Schema Definition)

@wbeaton There's been some discussion regarding this in the Eclipse IDE WG.

As Ed mentioned, the WG maintains a list of the projects under its wing. However, this list was built statically at the start of the WG and is now out of date. As you can see, Eclipse Collections was accidentally left out and should be added. OTOH, if we look at Wayne's list, we immediately see that some projects are no longer contributing either such as the Eclipse Accessibility Tools Framework.

It'd be great for the WG if we could automatically create this list and keep it up to date. The list that we want to show is the projects that actively contribute to SimRel. We also identified - as Ed mentioned - that some projects do not contribute to SimRel, but are instrumental to its success. These are projects such as Nebula and Jetty. We'd like to see a second list to

• make sure those projects get their due credit and fame
• visualize what projects are essential to keep the SimRel going and proactively mitigate any potential issues there

If Ed's simrel.aggran can lead us to consistent lists on eclipse.org/releases, the Eclipse N&N, the Eclipse IDE WG's project list that would be amazing. OTOH, I think your idea of creating a central aggregated Eclipse Release endpoint has a lot of merit as well. Perhaps Ed's simrel.aggran could instead use that site instead of PMI REST endpoints, while you could use Ed's simrel.aggran deduction work to create the aggregated endpoint?

Finally, personally, it makes sense to me to keep the This release is part of ... for now. Oftentimes when I research an Eclipse project, the PMI is the first and only resource I encounter. Any information that enriches it is valuable to me.

Also, the Eclipse IDE WG's activity is ramping up and it seems to me that in the near future, we'd be very interested in seeing contribution information for Eclipse Projects. Be it commit frequency, committers, releases, or contributing companies. That part still seems very valuable to me and as a company or individual, it's one of the most prominent places where you're credited for your work. Of course the same is true here, GitHub and GitLab already provide excellent statistics, but a single place (or a static URL) to query information makes the process much easier for cross-project or SimRel related work.

mentioned in issue eclipseide.org#45 (closed)

marked this issue as related to #155 (closed)

Note that I tracked project release records and, for lack of a better place, recorded them here:

https://github.com/merks/simrel-maven/issues/7

I did not see the expected email from Wayne with his usual list and this page doesn't exist:

https://projects.eclipse.org/releases/2023-06

Note that I know of at least two projects that produced released without release records

• Eclipse Dynamic Languages Toolkit • 6.2 • 2022-12-07
• Eclipse Remote Application Platform • 3.24.0 • 2023-03-15

I did inform the project leads of this oversight.

marked this issue as related to eclipseide.org#67 (closed)

mentioned in issue eclipseide.org#67 (closed)

mentioned in issue eclipseide.org#72

mentioned in issue #216 (closed)

FYI @webdev. As @jograham states above, the information that we're talking about moving is currently used to create the new and noteworthy. Any changes that we make regarding how we track participation needs to be able to feed that.

Here's the current format of the API that we created on projects.eclipse.org that feeds that N&N page on the eclipseide.org website: https://projects.eclipse.org/api/simultaneous_release/2023-12

This could be as simple as someone from the SimRel project who contributes a file with all the new and noteworthy links to the eclipseide.org website repo...

Is there an issue open for this in the Eclipse Simrel org?