Delayed email notifications

Summary

Email notifications from (at least) gitlab.eclipse.org seem to be delayed.

Steps to reproduce

Trigger an event on gitlab.eclipse.org that will send an email notification (e.g. merge an MR or comment on an watched issue).

What is the current bug behavior?

Notification is sent with a delay of multiple hours.

What is the expected correct behavior?

Notification is sent immediately.

Relevant logs and/or screenshots

Email header excerpt below. I censored some information where I'm not sure if it sensitive. Looks like the delay happened somewhere between mail.eclipse.org/localhost and outlook.com.

Received: from xxxxAxxxx.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:1cf::22)
 by xxxxBxxxx.EURPRD10.PROD.OUTLOOK.COM with HTTPS; Wed, 11 Oct 2023
 13:43:58 +0000
Received: from xxxxCxxxx.eurprd06.prod.outlook.com (2603:10a6:20b:49d::29)
 by xxxxAxxxx.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:1cf::22) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.41; Wed, 11 Oct
 2023 13:43:56 +0000
Received: from xxxxDxxxx.eurprd04.prod.outlook.com
 (2603:10a6:20b:49d:cafe::4e) by xxxxCxxxx.outlook.office365.com
 (2603:10a6:20b:49d::29) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.27 via Frontend
 Transport; Wed, 11 Oct 2023 13:43:56 +0000
Authentication-Results: spf=pass (sender IP is aaa.bbb.ccc.ddd)
 smtp.mailfrom=eclipse.org; dkim=pass (signature was verified)
 header.d=eclipse.org;dmarc=pass action=none
 header.from=gitlab.eclipse.org;compauth=pass reason=100
Received-SPF: Pass (protection.outlook.com: domain of eclipse.org designates
 aaa.bbb.ccc.ddd as permitted sender) receiver=protection.outlook.com;
 client-ip=aaa.bbb.ccc.ddd; helo=mail.eclipse.org; pr=C
Received: from mail.eclipse.org (aaa.bbb.ccc.ddd) by
 xxxxDxxxx.mail.protection.outlook.com (10.167.16.37) with Microsoft
 SMTP Server id 15.20.6838.22 via Frontend Transport; Wed, 11 Oct 2023
 13:43:55 +0000
Received: from localhost (localhost [127.0.0.1])
	by mail.eclipse.org (Postfix) with ESMTP id D088820FB5C6
	for <my.email.here@company.host>; Wed, 11 Oct 2023 02:38:16 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eclipse.org; h=
	auto-submitted:list-unsubscribe:list-id
	:content-transfer-encoding:content-type:content-type
	:mime-version:subject:subject:references:in-reply-to:message-id
	:reply-to:from:from:date:date:received:received:received; s=
	dkim; i=@gitlab.eclipse.org; t=1697006294; x=1697870295; bh=Wy5O
	6L48a69NUOAHJJDiRC4gqrp0Bocoz9d2cjQRFFw=; b=Jv0y1GtKvo25hvsi87Hn
	o8cyF04hCOg0hh/+ed5v29h4D1sAKhH0hx2O837hwxEHwx9GhgPdo4g5zpbJcVmk
	k3ShrgKHYRLYfhFmx1qY85L0UcccOi0W3wOqCW6tsPLAWLj8QGQ5qF4YQArAE8eR
	QmePmfthFOR3QQkDl54QAfQ=
X-Virus-Scanned: amavisd-new at eclipse.org
Received: from mail.eclipse.org ([127.0.0.1])
	by localhost (mail.eclipse.org [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id YfliCBOigN1d for <my.email.here@company.host>;
	Wed, 11 Oct 2023 02:38:14 -0400 (EDT)
Received: from gitlab.eclipse.org (ns31420057.ip-141-94-141.eu [141.94.141.31])
	by mail.eclipse.org (Postfix) with ESMTP id 1DFE720FB5C1
	for <my.email.here@company.host>; Wed, 11 Oct 2023 02:38:14 -0400 (EDT)
Received: by gitlab.eclipse.org (Postfix, from userid 9000)
	id 96152B20034; Wed, 11 Oct 2023 06:38:13 +0000 (UTC)
Date: Wed, 11 Oct 2023 06:38:13 +0000

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

Delayed development workflow.