We are closing out this window for the time being as the repair attempts have failed and the system requires further diagnostics from the hosting provider. For now we can operate normally and this will be resolved in the next business day or sooner. Thank you for your patience and understanding! We'll re-open a new maintenance window once we have a new timeline from our provider in order to do another failover to bring everything back online again.
Posted Mar 24, 2017 - 07:40 UTC
Update
There is a repair in progress on the affected server and we're awaiting the result. For now everything is still operational.
Posted Mar 24, 2017 - 07:21 UTC
Update
Our hosting provider is still looking at the affected system and we don't have an estimated time of completion yet. For now everything is operational and stable.
Posted Mar 24, 2017 - 05:08 UTC
Update
We had a not so graceful failover, but have the site back online for the time being and DNS changes are processing normally while the server is being repaired. We will put the site back into maintenance mode once the system is repaired and back online so we can migrate everything back into place.
Posted Mar 24, 2017 - 03:00 UTC
Identified
We have been alerted to a hardware failure on our primary database system and need to power it down in order to perform diagnostics and repair on the system. Online repair attempts have been attempted and failed so we unfortunately need to perform a failover to allow the system to be shutdown. This will cause momentary outages in our production API and website as we perform the failover. DNS resolution will continue to function normally, but DNS changes and Secondary DNS may get delayed during the short failover process.