What happened?
On July 1st, 2026, some customers reported an issue to xMatters Customer Support where they were encountering authentication errors when accessing the web user interface.
Why did it happen?
This issue occurred because a database cluster reached 100% resource utilization, causing database requests to time out. An exceptional traffic pattern exposed a set of slow-running queries, which significantly increased query latency on the cluster. As latency increased, there was resource starvation and further degradation of the cluster's responsiveness, resulting in timeouts.
How did we respond?
As soon as Customer Support confirmed the issue, they engaged the Engineering teams to investigate. As designed, the system had restored itself before Engineering could implement any mitigation or recovery actions, but the teams were still able to identify and isolate the root cause.
What are we doing to prevent it from happening again?
To prevent this issue from reoccurring, the Engineering Team designed, developed, and tested a fix. They deployed the fix in the morning of Wednesday, July 8th, 2026, and it has been successfully implemented for all instances.