Summary of event: On May 25th, Doxy.me experienced a database issue that resulted in 27 minutes of unplanned downtime from 2:03pm to 2:30pm ET.
Business Impacts: Sessions in progress were not impacted. The incident resulted in providers being unable to log in for their sessions. In addition, patients who were not already in the queue were unable to check in to their providers waiting room.
Root cause: A data import file was processed internally. While it was being processed, a series of long-running secondary server processes were triggered. Unfortunately, these processes built up in number, consuming additional server resources until the servers were backed up.
How was the issue resolved: The file was able to finish processing and the system self recovered.
Preventative next steps: We are optimizing the the long-running processes which will help prevent this scenario from happening again. In the short term, we are no longer running large data imports during peak business hours.