On August 27, 2020, at approximately 12 AM Pacific, some customers reported an issue to xMatters Customer Support where they were unable to log into the latest version of the xMatters mobile app on Android. Customers who installed the 2.26.0 version of the Android app were being logged out of the app, and encountered either a "Host does not exist or is unavailable" or an "Invalid server certificate" error message when attempting to log back in. Customers who had not upgraded their apps to the latest version were not impacted.
This issue was caused by a bug introduced in the 2.26.0 version of the Android app. The release build process includes a step that adds code optimization supplied by a third-party vendor. The software points to several public certificate guarantors and is added as part of the final build before the app is released to the Google Play store. In this case, the third-party optimization software was pointing to incorrect guarantors. Although the app development team performed a final set of tests prior to release and after the app was optimized, the optimization code was pointed at development guarantors which were not affected by the bug. The app passed the testing phase and was promoted to public release.
As soon as customers began reporting issues with logging into the Android app, the xMatters development team began to troubleshoot the issue. After testing, they determined that the issue was not due to localized external factors and decided to roll back the app to the previous version (2.25.2), which would appear in the store as version 2.26.1. They recompiled the app and
uploaded it to the Google Play store, simultaneously turning off automatic upgrade prompts to help prevent anyone else from upgrading to version 2.26.0. Although a working version of the app was uploaded to the store at 3:10 AM, and the testing and verification process from Google typically takes less than two hours, the process has been slowed recently due to the impact of
COVID-19. Due to the delay in the verification process, xMatters Customer Support posted a status page update to inform customers of the issue and to help prevent any other updates before the fixed version was available. The updated 2.26.1 version of the app was released to the store at 10:15 AM.
To help prevent the issue from reoccurring, the development teams has implemented a second round of QA to the release process. Due to the difference between test guarantors and production guarantors, testers outside of the standard QA process will perform a final functionality test before applications are promoted to production. This will account for any future differences between production and test environments.
August 26, 2020
5:40 PM - xMatters Android app version 2.26.0 released to Google Play store
August 26, 2020
11:40 - PM Google publishes app to Google Play store
August 27, 2020
2:00 AM - First reports of login issues from customers
2:20 AM - xMatters Engineering identifies an issue with the app and initiates rollback procedures
3:08 AM - Rollback of app to previous version (2.25.2), published as 2.26.1 in the Google Play store
3:15 AM - App auto update notification disabled
8:49 AM - xMatters publishes status page update
10:29 AM -Version 2.26.1 verified and published to the Google Play store
If you have any questions, please visit http://support.xmatters.com