This incident is now resolved. We will prepare and publish a public post incident document on the DNSimple blog once we have compiled all of the information from the incident.
Posted Mar 02, 2021 - 21:05 UTC
Monitoring
All queries are now be handled properly across all regions. We will continue monitoring for the time being to ensure we have addressed all resolution issues.
Posted Mar 02, 2021 - 20:28 UTC
Update
The majority of queries are now be handled properly, however, we are still seeing occasional incorrectly cached results for queries sent to our DDoS defense layer. We are working to identify and address this caching issue.
Posted Mar 02, 2021 - 20:02 UTC
Update
We have stabilized DNS resolution except for ALIAS records, we are currently working on those. We will continue to provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 19:26 UTC
Update
We see a normalization of the traffic patterns and in most regions resolution is working properly again. We are still working on US east coast region, to implement a fix there. We will continue to provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 18:46 UTC
Identified
We've completed the rollout of the mitigation to one region and the load is back to normal. We’re rolling out the same change to the other two affected regions.
Posted Mar 02, 2021 - 18:15 UTC
Update
We relentlessly continue working on identifying the cause of the SERVFAIL responses. We are implementing different measures to mitigate the traffic patterns that are causing this. We will continue to provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 17:44 UTC
Update
We continue working on identifying the cause of the SERVFAIL responses. We will continue to provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 17:07 UTC
Update
We continue to investigate sources of the high traffic volume that is leading to intermittent SERVFAIL responses on affected regions. We will provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 16:35 UTC
Update
We are experiencing a high query load in some regions which results in SERVFAIL on most DNS queries within the affected regions. We will provide updates as soon as we have them, but no later than 30 minutes from now.
Posted Mar 02, 2021 - 15:58 UTC
Update
We are all hands on deck continuing investigating the cause of the problem. You can expect the next update within the next 30 minutes.
Posted Mar 02, 2021 - 15:27 UTC
Update
We are continuing to investigate this issue.
Posted Mar 02, 2021 - 14:49 UTC
Investigating
We received some reports about resolution failures in certain regions. We are currently investigating this issue.