New API endpoint for Eclipse repositories

Summary / Short description

Wayne has developed a script that lists all repositories for all projects, available here: https://www.eclipse.org/projects/export/repositories.csv.php

This is used (internally) in a few places, from mass analysis of projects/repos to the Eclipse Metrics program.

Category

Detailed description if needed

There are more detailed information available in:

• https://gitlab.eclipse.org/eclipsefdn/emo-team/internal/eclipse-metrics/-/issues/6
• https://gitlab.eclipse.org/eclipsefdn/emo-team/internal/eclipse-metrics/-/issues/8

/cc @wbeaton

Is it possible for you to use the following endpoint?

https://projects.eclipse.org/api/projects

It contains all the projects repos.

AFAICT, not always. Few examples:

• Eclipse BlueChi: https://projects.eclipse.org/api/projects/automotive.bluechi
• Eclipse Kuksa: https://projects.eclipse.org/api/projects/automotive.kuksa
• Eclipse XFSC: https://projects.eclipse.org/api/projects/technology.xfsc

I'd guess that, in recent projects, when the org is set, repos are not listed.

For the records (and the corner cases):

This script lists all repositories for a project, not only a github or gitlab org (as in the PMI). In cases where a project (e.g. Eclipse Kuksa) has repos both in the GitHub Eclipse org (e.g. https://github.com/eclipse/kuksa.val) and in their own project org (e.g. https://github.com/eclipse-kuksa), it lists them all.

The script also lists repositories that are empty, and repositories that are used for websites (in case you wondered).

Where does the data come from for repositories.csv.php?

Start here:

https://gitlab.eclipse.org/eclipsefdn/emo-team/project-services/-/blob/master/capture/php/util/Project.class.inc?ref_type=heads#L204

And also in Java:

https://gitlab.eclipse.org/eclipse/technology/dash/eclipse-project-code/-/blob/main/src/main/java/org/eclipse/dash/ip/project/code/RepositoryService.java?ref_type=heads#L29

This is the problem with the Projects endpoint... I've already had to interpret and extrapolate from it twice. It would be good to have one single endpoint that gives us all of the information.

@epoirier The code that I've provided uses GitHub APIs to identify repositories in GitHub orgs, the GitLab API to identify repositories (recursively) in GitLab groups, and then adds repositories that are listed individually. The Java version uses actual libraries to do this. The PHP version makes REST calls directly and interprets the JSON results.

I have tried to use the excluded groups feature where I can, but you'll likely notice that I've had to hardcode a few exceptions to avoid, for example, including the OpenJDK repositories that Adoptium clones as "project repositories" (and similar for Oniro). I need to try hard to configure projects using the exclusion features so that I can remove the hard coded content.

You'll also notice that the Java implementation hardcodes fewer exceptions. This is because the context in which that implementation is used never encounters Adoptium or Oniro.

There's some more discussion here:

https://gitlab.eclipse.org/eclipsefdn/emo-team/internal/eclipse-metrics/-/issues/6

The script also lists repositories that are empty, and repositories that are used for websites (in case you wondered).

FYI, I started experimenting with some code that tries to determine whether or not a repository has "interesting" content. A repository that only contains a README, LICENSE, ... is not interesting (in an initial contribution review context, we would skip an uninteresting repository until it becomes interesting). I'll likely push that update later today.

It might be helpful to include some metadata along with the list of repositories.

e.g., it would be helpful to know whether or not the repository isEmpty or hasInterestingContent

@wbeaton and I spoke about this project today.

I suggested that we extend our existing Project API to ensure that all GitHub and Gitlab repos URLs are included in the body. This data should be available to us via the Dash database.

We used to track all the GitHub and GitLab repos in the PMI but over time, the purpose of those fields changed when we started to track GL groups and GH orgs.

This should not replace the existing GitLab and GitHub fields. The idea here is to merge these 2 datasets and remove duplicates.

These changes may have an impact on our sync script. We will need to ensure that @zacharysabourin and @malowe are involved before we deploy any changes to our API.

I plan to make this an MBO for us to implement this functionality in D9 in Q1. It will go live when the PMI D9 is ready for production.

I created the following MR to add the missing repos to the github and gitlab arrays on the project API using the data from the dash db.

https://gitlab.eclipse.org/eclipsefdn/it/websites/drupal/eclipsefdn/-/merge_requests/265

@epoirier Can you share a link from staging to allow @wbeaton and @bbaldassari2kd to review the changes and provide feedback on the format if they have any special requirements?

Here is the url to the projects endpoint for the Drupal 10 website:

https://projects-staging-d9.eclipse.org/api/projects

Hi, unfortunately, I cannot access this endpoint. My Eclipse credentials do not seem to work.

@bbaldassari2kd The credentials for our staging server are:

username: testuser password: plaintext

changed due date to March 31, 2024

added to epic &15 (closed)

assigned to @theodoreb

added statetodo label

added MBO label

added prioritymedium label

added severitylow label

changed milestone to %PMI 2.0

assigned to @epoirier and unassigned @theodoreb

added statereview label and removed statetodo label

mentioned in issue eclipsefdn/it/api/eclipsefdn-api-common#68 (moved)

Closing this issue, the MR that I created to add the missing repos has been merged.

closed