Issue Discovered - Service disruption in North America and Europe with Secondary Voice Provider
Incident Report for xMatters

What happened?

On Tuesday, October 31, 2017, xMatters monitoring alerted the Client Assistance team to a problem with some voice notifications not being delivered to recipients in North America and Europe. Other notification methods such as email, SMS, and push messages to the xMatters mobile apps were working as expected. The issue impacted clients running a legacy integration only - specifically those using a voice-over-IP delivery system (SIP). 

Why did it happen?

This issue occurred when the provider responsible for delivering voice notifications over SIP detected fraudulent behavior on their system and disabled the service without alerting xMatters.

How did we respond?

As soon as the xMatters monitoring tools alerted them to the issue, Client Assistance initiated a Severity-1 process and posted a bulletin on the xMatters status page. The incident response teams began simultaneously investigating the underlying cause and working to restore services for clients. They quickly determined that the issue was due to a problem with the voice provider, and raised a Priority-1 ticket with the provider to investigate the cause of the voice delivery failures. The provider identified that the triggers designed to prevent fraudulent activity on the account had detected suspicious behavior and automatically disabled the account. The provider immediately reactivated the account, and the incident teams confirmed that the system was delivering voice notifications again.

What are we doing to prevent it from happening again?

To prevent this issue from reoccurring, xMatters is reviewing all threshold levels and alerting set by the voice provider to ensure they align with xMatters' tolerance of risk for fraudulent activity. xMatters also recommends that all clients still using SIP work with our Client Success team to explore options for upgrading to the latest technologies provided by xMatters. This will allow impacted clients to take advantage of the latest and greatest features, and automatically provide redundancy for voice notification service providers.

Timeline:

2017-10-31 7:45PM - xMatters Client Assistance alerted to a possible issue with voice notifications impacting the secondary voice provider

2017-10-31 8:45PM - Initial investigation determines the issue is related to the voice provider

2017-10-31 8:48PM - Severity-1 process is initiated by xMatters Client Assistance

2017-10-31 8:51PM - Client Assistance receives first reports of failed voice notifications by clients

2017-10-31 9:00PM - Opened P1 ticket with voice provider

2017-10-31 9:24PM - Posted Support Bulletin - http://status.xmatters.com/incidents/xjzbh3jq9h9y

2017-10-31 10:05PM - Voice provider re-enables the account (automatically disabled due to suspected fraudulent behavior)

2017-10-31 10:07PM - Confirmed issue is resolved; voice notifications being delivered

If you have any questions, please visit http://support.xmatters.com

Posted 3 months ago. Nov 07, 2017 - 14:49 PST

Resolved
The issue has been addressed by the provider, and all services have been restored. Thank you for your patience while we addressed this matter.
Posted 3 months ago. Oct 31, 2017 - 22:15 PDT
Update
The issue is related to an upstream provider service disruption. We are continuing to work with the secondary provider to resolve this issue. We will provide updates as they become available.
Posted 3 months ago. Oct 31, 2017 - 22:01 PDT
Identified
The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.
Posted 3 months ago. Oct 31, 2017 - 21:30 PDT
Investigating
The xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in North America and Europe. We are currently investigating an issue with voice notifications for clients running legacy integrations. We will provide an update as information becomes available.
Posted 3 months ago. Oct 31, 2017 - 21:24 PDT
This incident affected: Europe, Middle East, and Africa (Voice Notifications) and North America (Voice Notifications).