DNS outage
Incident Report for DNSimple
Resolved
The redirector service is now restored and is answering requests. Much of the team is now getting some well deserved sleep and we will continue to work through the backlog of support requests. We'll have a post-mortem posted very soon after we conduct more research with our network provider. Thank you very much for your patience and support!
Posted about 4 years ago. Dec 02, 2014 - 08:17 UTC
Update
DNS is resolving normally along with our API and Website. We are restoring the redirector service right now and working through the back log of support requests.
Posted about 4 years ago. Dec 02, 2014 - 07:57 UTC
Monitoring
We are no longer experiencing a DDoS and are performing cleanup tasks now.
Posted about 4 years ago. Dec 02, 2014 - 07:37 UTC
Update
Our name servers are now resolving UDP traffic. We are still investigating issues with some third-party resolvers returning SERVFAIL responses to queries. We will continue to update here when we have more information.
Posted about 4 years ago. Dec 02, 2014 - 06:15 UTC
Update
We are beginning to see name servers resolve across all data centers.
Posted about 4 years ago. Dec 02, 2014 - 05:01 UTC
Update
We are continuing to investigate all avenues for mitigating the attack.
Posted about 4 years ago. Dec 02, 2014 - 03:52 UTC
Update
The attacks seem to be ramping up again across all data centers. We are working on further mitigation strategies.
Posted about 4 years ago. Dec 02, 2014 - 02:42 UTC
Update
We are seeing renewed attacks in our west coast data center.
Posted about 4 years ago. Dec 02, 2014 - 02:31 UTC
Update
Most of our US data centers are back online, but we are still working to restore service on our Tokyo and Amsterdam servers so you may still see resolution errors if you are on the US West coast or other non-US locations.
Posted about 4 years ago. Dec 02, 2014 - 02:16 UTC
Update
We have been able to bring our name servers online in the US and are actively working on bringing our other data centers back online.
Posted about 4 years ago. Dec 02, 2014 - 01:02 UTC
Update
Some name servers are coming back online, but we are still dealing with a partial DNS outage. We are working hard to resolve it now.
Posted about 4 years ago. Dec 02, 2014 - 00:14 UTC
Update
The ETA from our network provider has been extended another 30 minutes while they work on getting the new higher capacity device wired up.
Posted about 4 years ago. Dec 01, 2014 - 23:25 UTC
Update
30 minute ETA from our network provider to begin scrubbing traffic in a location with capacity.
Posted about 4 years ago. Dec 01, 2014 - 22:45 UTC
Update
For everyone following updates, we changed the name of our Twitter Status account to @dnsimplestatus
Posted about 4 years ago. Dec 01, 2014 - 22:18 UTC
Update
Our network provider is still working on increasing our capacity to mitigate the incoming attack. We'll update with an ETA when we get confirmation from our provider.
Posted about 4 years ago. Dec 01, 2014 - 21:35 UTC
Update
Our network provider is presently working on increasing capacity to deal with the flood of traffic. We will continue to post updates as we have more details.
Posted about 4 years ago. Dec 01, 2014 - 20:42 UTC
Update
We are still working with our network provider to mitigate the volume of traffic and will post more details here as we have them.
Posted about 4 years ago. Dec 01, 2014 - 20:21 UTC
Update
This attack is volumetric in nature. While we have measures in place for DDoS attacks, they have been overwhelmed by the volume of this attack.
Posted about 4 years ago. Dec 01, 2014 - 20:03 UTC
Identified
We have identified a DDoS attack and are working with our network provider to mitigate it.
Posted about 4 years ago. Dec 01, 2014 - 19:45 UTC
Investigating
We are seeing DNS connection timeouts on all systems. Currently investigating.
Posted about 4 years ago. Dec 01, 2014 - 19:19 UTC