Summary of event: From 12:45 EST to 13:10 EST on 12/01/2020, doxy.me customers experienced server errors, inability to log in, or load the page.
Root cause: There was an internal service that temporarily needed 70ms instead of 1ms for its requests due to a failover. While the difference between 70 ms and 1 ms may seem small, this internal service receives a large number of requests in the middle of the day which caused a memory shortage.
How was the issue resolved: We routed all traffic to another set of servers. We moved the traffic back and increased server capacity and memory to accommodate the additional requests on 12/02/2020.
Preventative next steps: In the next several weeks, we will fail back to the internal service that had the issue and the load will be reduced. The event allowed us to refine our scaling capabilities to respond better to service or server failovers.