The issue causing increased load on our database, which impacted the Abnormal Portal Threat Log and SOAR API endpoints, has been resolved.
Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints
Severity Level: Major
Incident Start Time: May 31, 2024, 10:00 UTC
Resolution Timeline: May 31, 2024, 21:00 UTC
Current Status: The affected services are now fully operational. We have verified that the database load has normalized, and all systems are functioning as expected.
Next Steps: Our team will continue to monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future.
Thank you for your patience and understanding throughout this incident.
Posted May 31, 2024 - 15:00 PDT
Update
Our team continues to work on resolving the increased load on our Abnormal Portal / Threat Log and SOAR API endpoints. While we have made progress, the issue remains unresolved.
Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints
Severity Level: Major
Incident Start Time: May 31, 2024, 10:00 UTC
Current Status: Our Engineering team is actively implementing solutions to mitigate the increased load. We are committed to resolving this issue as quickly as possible and will keep you updated on our progress.
Incident Update: Our team continues to work on resolving the increased load on our database caused by the recent Azure Sentinel integration change.
Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints
Severity Level: Major
Incident Start Time: May 31, 2024, 10:00 UTC
Current Status: Our Engineering team is actively working on implementing a solution. Unfortunately, the issue is still not fully resolved. We are making progress and will provide further updates as soon as possible.
On May 31, 2024, at approximately 10:00 UTC, a change was released to the Azure Sentinel integration for all customers using the Abnormal Security. This change altered how Sentinel queries our API by implementing a different date range filter, resulting in queries over a larger time range. Consequently, this led to an increased load on the database powering all threat information in both the SOAR API and the Portal, causing higher latency in the Threat Log and errors in Azure Sentinel calls due to a broken date range filter.
There is no impact on Email Detection and Remediation.
Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints
Severity Level: Major
Incident Start Time: May 31, 2024, 10:00 UTC
What We're Doing: Our Engineering team has identified the root cause and is implementing a solution. We expect the database performance to improve in the next hour.