TLDR: IES, AMB, and EPR experienced delayed remediation by several minutes during a ~1h incident period starting at 15:40 UTC on 1/3/24.
What Happened The live scoring system system experienced instability between 15:40 and 16:05 UTC. This led to delayed remediation for attack and graymail messages from 15:40 to 16:45 UTC by several minutes. All systems are stable now and all messages are processing at normal processing times as of 16:45 UTC. All messages delayed during the period of instability have been remediated as of 16:45 UTC. No customer action is needed.
Root Cause Excessive load on a database outside of the intended live scoring flow led to live message remediation delay. We are investigating the root of the excessive load and will ensure we address the scale aspect of the systems concerned.
Follow Ups Root cause investigation for excessive database load.
Posted Jan 03, 2024 - 09:16 PST
This incident affected: Inbound Email Security (IES), Email Productivity (EPR), and AI Security Mailbox (AISM).