Blank white page when loading doxy.me
Incident Report for Doxy.me LLC
Postmortem

Root cause analysis

On Tuesday, June 16th, 2020, we deployed an update to the doxy.me website. This change had several enhancements, optimizations, and technical changes. The most prominent feature deployed that day was our consolidation of SMS notifications to a US short-code.

One of the technical changes (not related to SMS) added an unoptimized SQL query that queried an unindexed field in the database. We deployed the change very early in the morning, but once we started seeing a lot of traffic, the unoptimized query abruptly slowed down our database and made our website unresponsive.

This unresponsiveness of our database made it so both providers and their patients would see a blank white screen instead of loading the correct page.

The short term mitigation was to roll back the particular change. The medium-term resolution was to reimplement the change with the correct indexing.

Long term we will be doing several things. First, we implement a new process that requires advanced scrutiny for changes that involved database queries. We will eventually automate load testing of any change during our software development lifecycle. And finally, we will be adding an additional screen that will display besides just a blank white page. This page will be more informative and help direct the customer or the patient to our status page so they can be informed of what is going on.

We realize the importance doxy.me plays to not only your business but to the health and wellbeing of your patients. We strive to deliver a good product, with great service. We can always be better and we thank you for your continued support.

Posted Jun 18, 2020 - 23:10 EDT

Resolved
The fix looks like it was successful and doxy.me is fully operational. We will mark this as resolved and provide an RCA later today.
Posted Jun 16, 2020 - 09:23 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 16, 2020 - 09:06 EDT
Update
The fix was implemented and deployed, we are just waiting for changes to take effect.
Posted Jun 16, 2020 - 09:03 EDT
Update
The update to the servers is taking longer than expected. Please allow for a few more minutes. We will post another update in 10 minutes.
Posted Jun 16, 2020 - 08:51 EDT
Update
We are continuing to work on a fix for this issue. Estimated time to resolution: ~10-15 more minutes.
Posted Jun 16, 2020 - 08:36 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Jun 16, 2020 - 08:26 EDT
Investigating
We are currently getting reports of not being able to load doxy.me. We are investigating and will report back in a few minutes.
Posted Jun 16, 2020 - 08:22 EDT
This incident affected: Doxy.me API.