On Tuesday, June 16th, 2020, we deployed an update to the doxy.me website. This change had several enhancements, optimizations, and technical changes. The most prominent feature deployed that day was our consolidation of SMS notifications to a US short-code.
One of the technical changes (not related to SMS) added an unoptimized SQL query that queried an unindexed field in the database. We deployed the change very early in the morning, but once we started seeing a lot of traffic, the unoptimized query abruptly slowed down our database and made our website unresponsive.
This unresponsiveness of our database made it so both providers and their patients would see a blank white screen instead of loading the correct page.
The short term mitigation was to roll back the particular change. The medium-term resolution was to reimplement the change with the correct indexing.
Long term we will be doing several things. First, we implement a new process that requires advanced scrutiny for changes that involved database queries. We will eventually automate load testing of any change during our software development lifecycle. And finally, we will be adding an additional screen that will display besides just a blank white page. This page will be more informative and help direct the customer or the patient to our status page so they can be informed of what is going on.
We realize the importance doxy.me plays to not only your business but to the health and wellbeing of your patients. We strive to deliver a good product, with great service. We can always be better and we thank you for your continued support.