TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline.
Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 09:00 UTC: Processing for messages that were delayed during the incident timeline from 00:00 - 02:15 UTC is complete. The incident is now fully resolved. 02:15 UTC: Live processing for message remediation has recovered for all customers going forward. 01:55 UTC Other required backend services required to ensure message remediation success are healthy. The team is increasing traffic volume to the relevant backend systems to regular message processing volume. 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline.
Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 02:15 UTC: Live processing for message remediation has recovered for all customers going forward. 01:55 UTC Other required backend services required to ensure message remediation success are healthy. The team is increasing traffic volume to the relevant backend systems to regular message processing volume. 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB
What We're Doing: The remediation actions that were delayed are currently being processed Estimated Resolution Time: December 08, 2023, 02:15 UTC
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline.
Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB
What We're Doing: The team is restoring regular message processing traffic volume to the relevant backend systems. Estimated Resolution Time: TBD
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline.
Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 01:10 UTC: Current status - investigation of solution for delayed remediation. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB
What We're Doing: We are currently investigating the appropriate solution to solve the degradation. Estimated Resolution Time: TBD