Create notification if a PR fails to revalidate after a number of times
When attempting to debug a different problem, the logs were overloaded with unrelated errors for PR's that were failing to validate properly. We need to create a mechanism for us to receive notifications when errors are long-standing so that we can address them as they become problematic. I'm thinking we can create a list of addresses that receive the message when a certain threshold is met by the revalidation process.
I'm thinking to start with, I can be the sole person on the list, with a threshold of 100 runs before sending a notification. While that is a lot, that is less than a half day of consistent failures, so it should notify us reasonably fast when there is a broken PR in the system for whatever reason. We can make that threshold configurable as well so that we can fine-tune this in the future easily.
@cguindon any thoughts/concerns about this?