Increased load and slow load time
Incident Report for LLC

On March 16, 2020 at 10:28 EDT, saw an increase in traffic resulting in a server load increase and a degradation in response times across the service. While the service continued to sign up nearly 2,500 providers every hour, the team acted swiftly to mitigate the issue by updating key elements of the infrastructure starting with the supporting database and elements of our API. This did not alleviate the problem completely and required additional steps such as utilizing an additional server to offload critical processes and further load balance requests. At this point, service resumed for the vast majority of providers and their clients. As a result of today’s efforts, has redoubled efforts to ensure all backup infrastructure is available and ready to deploy in a matter of minutes and has additionally identified potential improvements to both infrastructure and processes.

Needless to say, has seen significant interest from providers in need of a simple and secure solution to continue to provide care and assistance to their clients. The entire team takes the responsibility of maintaining our telemedicine service with the utmost importance and we are striving to meet the increase in growth with the best we can throw at it. Our team has grown exponentially almost overnight and we have never been more proud of being able to provide a free telemedicine solution in a time when it is needed most. While we understand the events of earlier today are stressful and challenging for all providers and their patients, we are encouraged that today nearly 20,000 providers signed up to use to provide critical care in a time of global need.

We thank you all for your understanding and we look forward to continuing to empowering our providers!

Posted Mar 16, 2020 - 17:53 EDT

We have resolved this issue and will continue to monitor the system as normal. A post mortem review will be added shortly with additional details. Thank you.
Posted Mar 16, 2020 - 17:51 EDT
We are continuing to monitor for any further issues.
Posted Mar 16, 2020 - 14:22 EDT
Things are looking better operationally speaking. But still going to keep this open for a while longer.
Posted Mar 16, 2020 - 14:12 EDT
A fix has been implemented and we are monitoring the results.
Posted Mar 16, 2020 - 13:42 EDT
We are continuing to work on a fix for the capacity constraints we are seeing.
Posted Mar 16, 2020 - 12:20 EDT
We are continuing to work on a fix for this issue.
Posted Mar 16, 2020 - 11:40 EDT
We're are implementing emergency maintenance. There will be 5 minutes of complete downtime.
Posted Mar 16, 2020 - 11:17 EDT
We are continuing to work on a fix for this issue.
Posted Mar 16, 2020 - 10:49 EDT
We've identified an issue that is making page loads slow, we are implementing a fix now.
Posted Mar 16, 2020 - 10:28 EDT
This incident affected: API.