Github sync removes user that can't be found due to server error

Earlier today, the accounts API went down due to internal errors, and users were being mass removed from the API. When the API responds with 500, we should skip processing the user either way as it indicates bad server state rather than a bad user state.

added statewip label

assigned to @malowe

mentioned in merge request !244 (merged)

mentioned in commit 8ada2646

I few people, including myself, got removed from a list of projects in GitHub.

Will that be corrected automatically? Should people reach out and let you know?

On my side, I was removed from 3 project, then re-added ~2 or 3 hours later without reporting anything.

I found those e-mail too. Reading slack it looked like people just did not know what the status is.

We had sent out a few emails regarding the status, but it looks like it was sent to a less-popular mailing list rather than the main one. We're addressing that internally at the moment, but I'll clarify the status here as well. The service was restored 2h after the incident and was caused by some errors in some of our internal services and routing. We posted to our status page as the incident evolved, and can be seen on the specific incident page.

I wasn't aware of the status page. But that's just me. I think it is super important, and may be worth advertising a bit more

However, the text in the status message doesn't say anything about "what's next" in context of the lost permissions. I think it would be good to have word or two in there of what people that lost permissions should expect, or if there is something (or nothing) to do.

I wasn't aware of the status page. But that's just me. I think it is super important, and may be worth advertising a bit more

We do what we can to talk about the status page in our incident messaging on mailing lists, as well as on issues like this to best raise awareness. We also have it linked in the footer of our websites to increase visibility.

However, the text in the status message doesn't say anything about "what's next" in context of the lost permissions. I think it would be good to have word or two in there of what people that lost permissions should expect, or if there is something (or nothing) to do.

This is where the mailing list is important, as when there are messages like this, they will end up either on the status page, or sent out in an email like the one sent yesterday morning once we have a resolution to the issue. In this case, we sent the email once we confirmed the cause and effect of the outage, and what steps are needed if any!

mentioned in merge request !245 (merged)

mentioned in commit cd646015

closed with commit c315c560

closed with merge request !245 (merged)

mentioned in commit c315c560