First off, I'd like to apologize for our outage on Monday May 1, at approximately 18:10 EDT. We appreciate your business and know how important this service is to you and your patients.
While these outages are rare, the past week has been especially spotty given our track record over the past 5 years. Culminating in a larger issue Monday which I will go into more detail here.
At a high level there were three distinct issues.
For a while we were specific on what version of a piece of software we use. But with automatic updates our customers would get the most updated version, which results in quicker bug fixes and the most up-to-date features. The downside is if there is a bug customers can be affected without warning.
After identifying the issue, we ran into several issues while implementing the fix. We have an automated system that builds, tests, and updates the website. This is great as it automates a lot of manual work and can reduce errors from manual processes. The downside is when we made our change, the tests would not pass and our update could not continue.
The third problem, is that the process is slow. It can take anywhere from 10-20 minutes depending on the changes to go through this automated update process. And after a few broken tries to deploy our fix, the amount of time it took to get the fix live on Doxy.me was substantial.
While we understand this incident was painful for you, our customers, we did learn a few valuable lessons that we will be implementing over the next month to help prevent this type of issue from happening again.
Thank you again for being a Doxy.me customer and we look forward to making improvements to our system to help avoid and mitigate these types of issues in the future.