GitHub auto-close does not work

Summary

We want GitHub to automatically close issues if they are referenced from commit messages as documented here.

In most cases this does not work in the https://github.com/eclipse-apoapsis/ort-server repository and it is totally unclear to me why. I could not find any documentation that this feature could be disabled, and I could not find any reports of other GitHub users having this issue.

In other projects I contribute to, like https://github.com/oss-review-toolkit/ort, we never had any problems with auto-closing issues from commit messages.

Could this be caused by some configuration you enforce on GitHub organizations? If not, could you reach out to GitHub support for help?

We collect some of the affected issues here: https://github.com/eclipse-apoapsis/ort-server/issues/868

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

I can only imagine that this problem is related to the use of a merge queue. This is also the difference between ort and ort-server afaict.

added IT-prioritymedium IT-severitynormal service:GitHub statetodo team:releng labels

Yes, maybe it's related to that. According to the docs the trigger should be the merge to the main branch, so the merge queue should not matter, but maybe there is a bug.

I could not find anything related to the problem that GitHub does not close referenced issues from PRs. It would be good if you could run a tests with and without merge queue and see if there is a connection. If we gathered more data points we can certainly open a ticket with GitHub support, though that might be unsatisfactory from experience.

Ok, I will try if I can reproduce the issue in another repository with merge queue enabled.

@netomi I could verify that the issue is indeed related to the merge queue. I have created https://github.com/oss-review-toolkit/autoclose-test for testing and checked the following scenarios:

1. Push directly to main with "Fixes #1 (closed)" in the commit message works.
2. Merge a PR without merge queue with "Fixes #2 (closed)" in the PR description and commit message works.
3. Merge a PR without merge queue with two commits with "Fixes #3 (closed)" and "Fixes #4 (closed)" in the commit messages but not in the PR description works.
4. Merge a PR via merge queue with "Fixes #5 (closed)" in the PR description and commit message works.
5. Merge a PR via merge queue with two commits with "Fixes #6" and "Fixes #7 (closed)" in the commit message but not in the PR description does NOT work.

You can see that issues 6 and 7 are still open, even if GitHub recognized that there are commits referencing them: https://github.com/oss-review-toolkit/autoclose-test/issues?q=is%3Aissue%20

The reason why 4 works is that if the issue reference is in the PR description GitHub links the PR to the issue. However, this link is not generated when the issue references are only in the commit messages. With out merge queue this still works as documented as shown in scenario 3.

So this is clearly a bug in GitHub that can easily be reproduced, now that we know how.

that is quite some extensive test, ty. I assume you did the merge queue test with merge_method: rebase? The method itself will probably make no difference.

So is this something that you can build upon, i.e. make sure that the PR description references the issues that should be closed? Maybe a workflow could be added to check if the description includes an issue link and issue a warning if this is not the case?

Reporting that to GH is an option, but it might take a while for them the react on it if ever.

I assume you did the merge queue test with merge_method: rebase?

Correct.

Reporting that to GH is an option, but it might take a while for them the react on it if ever.

I will discuss a workaround with the team, but it would be great if you could report the issue to GitHub anyway. I could do it as well, but I guess EF is an enterprise customer, so you should get more attention than if I would report it as an individual user?

alright will open a support ticket with them.

so I created the following ticket: #3207740

In the end I did create it for the eclipse-apoapsis project and it says standard support for that org. Every member of an org can create a support ticket for a specific org afaict. Lets see what they answer.

For the workaround I could not find a way to change the issues linked to a PR via the API, so the best I could come up with is a workflow that adds a comment to the PR if the issue references in the commit messages and the issues linked to the PR do not match: https://github.com/eclipse-apoapsis/ort-server/blob/main/.github/workflows/check-issue-links.yml

looks good, I found this though:

https://stackoverflow.com/a/72967359

using the graphql api you can get a list of issues that will be closed by a PR. Not sure if this would be helpful or if it works in this case.

stupid me, I should have looked at the workflow before, you already do what I found.

thats what I got now as a follow up:

Hello Thomas,

Thank you for your patience.

In this case I want to make sure that this isn'ta commit message issue.

As per the 2 commits you linked me here the commit messages for both of these are as follows:

Fixes #3 with PR.

Fixes #5 with MQ.

Whereas from your example here: https://github.com/eclipse-apoapsis/ort-server/issues/868

The commit message was Fixes #174.

Now I want to make sure this isn't just a mistake where the commit message . after the number is obfuscating the linkage.

Can you please attempt to reproduce this with a commit message only in the merge queue where a . does not immediately following the issue number that is attempting to be linked?

If you can please provide the PR and issue after that I will be happy to escalate this to my engineering team for further review!

Regards, Vincent GitHub Enterprise Support

so he misses the point that the commit message seems to be ignored in case of a merge queue. There are 2 examples, one where it works, one where not, and the commit message follows the same pattern.

The case where it actually works, the fixes message is in the PR description, while for the one where it does not work it is not.

I will answer something this line.

Hello Thomas,

Thank you for your patience.

I have created an internal issue here with my engineers to investigate this behavior. Once I have a response I will be sure to let you know.

Regards, Vincent GitHub Enterprise Support

@netomi Thanks for the update! I hope they consider a fix for this.

Hello Thomas,

My engineers are taking a look into this here at this time, but they have advised in the meantime as a workaround to utilize the description comments in order to close these linked issues.

Once I have more details I will be sure to let you know.

Regards, Vincent GitHub Enterprise Support

I answered that you already have a pretty neat workaround in place to ensure that closing references between commits and PR are in sync.

Hello Thomas,

Our engineering team has indicated that this work will not be completed in the near-term.

With that in mind I'm going to update the status of this support request as Solved. However, please note that the status of Closed or Solved is simply a state within our support system, and does not necessarily reflect the state of the internal issue with our engineering team.

Please keep an eye on the GitHub changelog and our social media feeds for the latest announcements about new features and fixes. You can also browse the GitHub public roadmap which is a project board that provides a high level overview of the current developments and their planned release schedule.

I understand that this is not an ideal response. However, I can assure you the issue is still being tracked in our system. If you have any additional instances of this that may help us raise the visibility and priority, please open a new ticket and we can add it to the issue while we continue to look at it. In the meantime, we will be closing out this ticket and we will send a follow up if we have any additional information in the future.

assigned to @netomi