What happened?

On Tuesday, October 31, 2017, xMatters monitoring alerted the Client Assistance team to a problem with some voice notifications not being delivered to recipients in North America and Europe. Other notification methods such as email, SMS, and push messages to the xMatters mobile apps were working as expected. The issue impacted clients running a legacy integration only - specifically those using a voice-over-IP delivery system (SIP). 

Why did it happen?

This issue occurred when the provider responsible for delivering voice notifications over SIP detected fraudulent behavior on their system and disabled the service without alerting xMatters.

How did we respond?

As soon as the xMatters monitoring tools alerted them to the issue, Client Assistance initiated a Severity-1 process and posted a bulletin on the xMatters status page. The incident response teams began simultaneously investigating the underlying cause and working to restore services for clients. They quickly determined that the issue was due to a problem with the voice provider, and raised a Priority-1 ticket with the provider to investigate the cause of the voice delivery failures. The provider identified that the triggers designed to prevent fraudulent activity on the account had detected suspicious behavior and automatically disabled the account. The provider immediately reactivated the account, and the incident teams confirmed that the system was delivering voice notifications again.

What are we doing to prevent it from happening again?

To prevent this issue from reoccurring, xMatters is reviewing all threshold levels and alerting set by the voice provider to ensure they align with xMatters' tolerance of risk for fraudulent activity. xMatters also recommends that all clients still using SIP work with our Client Success team to explore options for upgrading to the latest technologies provided by xMatters. This will allow impacted clients to take advantage of the latest and greatest features, and automatically provide redundancy for voice notification service providers.

Timeline:

2017-10-31 7:45PM - xMatters Client Assistance alerted to a possible issue with voice notifications impacting the secondary voice provider

2017-10-31 8:45PM - Initial investigation determines the issue is related to the voice provider

2017-10-31 8:48PM - Severity-1 process is initiated by xMatters Client Assistance

2017-10-31 8:51PM - Client Assistance receives first reports of failed voice notifications by clients

2017-10-31 9:00PM - Opened P1 ticket with voice provider

2017-10-31 9:24PM - Posted Support Bulletin - http://status.xmatters.com/incidents/xjzbh3jq9h9y

2017-10-31 10:05PM - Voice provider re-enables the account (automatically disabled due to suspected fraudulent behavior)

2017-10-31 10:07PM - Confirmed issue is resolved; voice notifications being delivered

If you have any questions, please visit http://support.xmatters.com