Server Errors and Slowness
Incident Report for LLC

Summary of event: From 12:45 EST to 13:10 EST on 12/01/2020, customers experienced server errors, inability to log in, or load the page.

Root cause: There was an internal service that temporarily needed 70ms instead of 1ms for its requests due to a failover.  While the difference between 70 ms and 1 ms may seem small, this internal service receives a large number of requests in the middle of the day which caused a memory shortage.

How was the issue resolved: We routed all traffic to another set of servers. We moved the traffic back and increased server capacity and memory to accommodate the additional requests on 12/02/2020.

Preventative next steps: In the next several weeks, we will fail back to the internal service that had the issue and the load will be reduced. The event allowed us to refine our scaling capabilities to respond better to service or server failovers.

Posted Dec 07, 2020 - 10:53 EST

The incident has been resolved.
Posted Dec 01, 2020 - 17:02 EST
Monitoring is back up at this time. We are continuing to monitor the situation.
Posted Dec 01, 2020 - 13:13 EST
We are currently investigating this issue.
Posted Dec 01, 2020 - 13:05 EST
This incident affected: webpages.