Issue Discovered - Service degradation in All Regions – Android Mobile App
Incident Report for xMatters
Postmortem

What happened?

On August 27, 2020, at approximately 12 AM Pacific, some customers reported an issue to xMatters Customer Support where they were unable to log into the latest version of the xMatters mobile app on Android. Customers who installed the 2.26.0 version of the Android app were being logged out of the app, and encountered either a "Host does not exist or is unavailable" or an "Invalid server certificate" error message when attempting to log back in. Customers who had not upgraded their apps to the latest version were not impacted.

Why did it happen?

This issue was caused by a bug introduced in the 2.26.0 version of the Android app. The release build process includes a step that adds code optimization supplied by a third-party vendor. The software points to several public certificate guarantors and is added as part of the final build before the app is released to the Google Play store. In this case, the third-party optimization software was pointing to incorrect guarantors. Although the app development team performed a final set of tests prior to release and after the app was optimized, the optimization code was pointed at development guarantors which were not affected by the bug. The app passed the testing phase and was promoted to public release.

How did we respond?

As soon as customers began reporting issues with logging into the Android app, the xMatters development team began to troubleshoot the issue. After testing, they determined that the issue was not due to localized external factors and decided to roll back the app to the previous version (2.25.2), which would appear in the store as version 2.26.1. They recompiled the app and
uploaded it to the Google Play store, simultaneously turning off automatic upgrade prompts to help prevent anyone else from upgrading to version 2.26.0. Although a working version of the app was uploaded to the store at 3:10 AM, and the testing and verification process from Google typically takes less than two hours, the process has been slowed recently due to the impact of
COVID-19. Due to the delay in the verification process, xMatters Customer Support posted a status page update to inform customers of the issue and to help prevent any other updates before the fixed version was available. The updated 2.26.1 version of the app was released to the store at 10:15 AM.

What are we doing to prevent it from happening again?

To help prevent the issue from reoccurring, the development teams has implemented a second round of QA to the release process. Due to the difference between test guarantors and production guarantors, testers outside of the standard QA process will perform a final functionality test before applications are promoted to production. This will account for any future differences between production and test environments.

Timeline:

August 26, 2020

5:40 PM - xMatters Android app version 2.26.0 released to Google Play store

August 26, 2020

11:40 - PM Google publishes app to Google Play store

August 27, 2020

2:00 AM - First reports of login issues from customers
2:20 AM - xMatters Engineering identifies an issue with the app and initiates rollback procedures
3:08 AM - Rollback of app to previous version (2.25.2), published as 2.26.1 in the Google Play store
3:15 AM - App auto update notification disabled
8:49 AM - xMatters publishes status page update
10:29 AM -Version 2.26.1 verified and published to the Google Play store

If you have any questions, please visit http://support.xmatters.com

Posted Sep 02, 2020 - 15:45 PDT

Resolved
The fix is now available in all Google Play stores.

If you are are experiencing login issues, please reinstall the xMatters Andriod app through the store, some customers may need to uninstall the xMatters app before installing the fix.
Posted Aug 27, 2020 - 11:23 PDT
Update
Some Google Pay stores have now posted version 2.25.2.

If you recently installed version 2.26.0 please go to the store and install version 2.52.2. If you did not install version 2.26.0 or are not experiencing login issues, you do not need to take any action.
Posted Aug 27, 2020 - 10:29 PDT
Monitoring
The latest version of the xMatters Android App (2.26.0) has impacted the ability to login for some customers. We've already identified the issue and rolled back the app version while we work on a fix. The rollback has been submitted to Android, but it can take some time for it to reach all stores around the world.

We have also turned off the prompts for in-app update and recommend that customers do not upgrade to version 2.26.0 at this time. If you have already updated your app to version 2.26.0, we recommend that you roll back to version 2.25.2 once it's available in the Google Play store.
Posted Aug 27, 2020 - 08:49 PDT
This incident affected: North America (Mobile App), Asia Pacific (Mobile App), and Europe, Middle East, and Africa (Mobile App).