Summary of event:
On March 23, doxy.me experienced intermittent service disruptions during a planned system upgrade. A combination of internal issues and a third-party service issue led to degraded performance throughout the day.
Incident Classification:
Service disruption caused by a combination of internal system changes and a third-party service issue. This was not a security breach.
Business impacts:
Some users experienced dropped calls, difficulty accessing History and Analytics, and issues with Custom Waiting Rooms.
Data Impact:
No evidence of unauthorized access, disclosure, or loss of customer data or PHI. Some data was temporarily unavailable during the incident but has been fully restored.
Root cause:
The disruption was caused by overlapping issues, including a third-party service not updating correctly, delays in loading History, and a migration issue affecting some settings.
How the issue was resolved:
Our team deployed fixes throughout the day and paused the upgrade to prevent further impact. Service returned to normal after all issues, including the third-party issue, were resolved.
Preventative next steps:
We are improving our system upgrade process, strengthening safeguards around third-party services, and enhancing monitoring to respond more quickly in the future.