Issue Discovered - Service disruption in North American Region – Integration Platform

Incident Report for xMatters

Postmortem

What happened? 

On June 12th, 2025, at approximately 11:29 AM Pacific, the xMatters internal monitoring tools alerted to an issue in the North America region with the xMatters integration platform. While the issue was in progress, some xMatters customers may have experienced workflows not executing correctly or signals not being accepted as expected. 

Why did it happen? 

The issue occurred because multiple Google Cloud and Google Workspace products experienced external API issues for about three hours between the hours of 10:49 AM to 1:49 PM Pacific. 

How did we respond? 

As soon as the xMatters monitoring tools reported an issue with the xMatters integration platform, the xMatters Customer Support team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly determined that the issues were due to problems with an upstream provider and immediately opened a support case for them to investigate. They continued to work with the provider throughout the incident and confirmed with their engineers as they recovered the underlying dependencies. 

What are we doing to prevent it from happening again? 

The xMatters Support and Engineering teams requested an RCA and prevention methods from the upstream provider. In their response, they confirmed that they will be modularizing their Service Control architecture to prevent failures from affecting the entire service.

Posted Jun 23, 2025 - 10:40 PDT

Resolved

We are currently seeing error rates down to normal levels and we are considering this issue resolved at this time
Posted Jun 12, 2025 - 15:12 PDT

Monitoring

At this time, we are seeing all services recovering. We will continue to monitor until we have confirmation from our upstream provider that they have fully resolved the issue.
Posted Jun 12, 2025 - 13:53 PDT

Update

We are continuing to work with our upstream providers. Their engineers have confirmed that the underlying dependency is recovered in most regions and all their engineering teams are actively engaged and working on service recovery.
We do not have an ETA for full service recovery.
Posted Jun 12, 2025 - 13:06 PDT

Update

We are still working with our upstream providers. Their engineers are continuing to mitigate the issue and we have confirmation that the issue is recovered in some locations. We do not have an ETA on full mitigation at this point. We continue to operate in a degraded state.
Posted Jun 12, 2025 - 12:34 PDT

Identified

The xMatters Incident Response team has identified the source of the issue with an upstream provider and is working with them on identifying a fix. We will update once a solution has been identified and implemented.
Posted Jun 12, 2025 - 12:04 PDT

Investigating

xMatters monitoring tools have identified a potential issue with the xMatters Integration Platform for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available.

If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.
Posted Jun 12, 2025 - 11:47 PDT
This incident affected: North America (Integration Platform).