Jeffrey I. Schiller
Networking, Security and Cavies

But for the Grace of the Domain Name Service...

In my last post where I discussed Amazon's EC2 problems and their resolution, I concluded with:

"All of this though does put an emphasis on where you procure domain name services. As ultimately your services (whether they be a mailbox or a website) are located by you and others via your domain name provider. If they fail, you may have a problem! I'll muse on this in another post."

Wow, was I prescient! Just a few weeks after that post my Domain Name Service (DNS) Provider suffered an outage. Whereas I was able to weather the storm at Amazon by using redundant instances, I had no such option with my DNS provider.

So what happened

The "core" business of a DNS provider, or "registrar" is to sell you an entry in a database. In my case an entry in the ".net" top level domain. The top level domain is actually maintained by someone else (most of the time), an entity called a "registry." The registries actually operate the servers that provide for the top level domains. As you might imagine, given that they are near the "root" of the Internet, these servers have to be very redundant both for performance and reliability. They are also distributed around the globe. These servers are very hard to successfully attack, and an attack would be catastrophic (and therefore demand significant resources in remediation).

However this database entry isn't enough. You need to have your own DNS servers to provide services to your domain. Because not many people are in a position to operate DNS servers, most DNS registrar's also provide DNS servers which you can use to host your own domain. These servers may be very reliable and redundant, or they may not!

In my case I opted to use the servers of my DNS provider, which has proven just fine for quite a few years. However a few weeks ago they suffered a Distributed Denial of Service (DDoS) attack against their infrastructure. I am pretty sure that I didn't offend anyone enough to trigger an attack, but at least one of their other customers did. And we all suffered.

Fortunately it was over a weekend and during a time where I had few users, so the impact was minimal, but definitely there. It was about 24 hours before the attack was mitigated.

So what did I do

Well, I learned from this. I decided to operate my own DNS servers (yes, you can do that!) on my Amazon instances. Now this does require some knowledge and skills, which not everyone has, but I do. At first blush I didn't feel comfortable with this solution, I had only recently dealt with the Amazon EC2 outage. Yet, if Amazon is out, so am I, so adding the risk of an Amazon failure also taking out my DNS didn't seem to bad. It also meant that I was no longer at risk of a DDoS attack on my DNS registrar having an impact on my services.

So far so good.

There was only one catch. When I used my provider's website to change my account to use my own DNS servers instead of theirs, they removed my entries from their DNS servers and updated the ".net" servers immediately. Unfortunately DNS records are cached around the Internet and the expiration of my records on their servers was sooner then the expiration of the records in ".net" that pointed to their servers. So I suffered a second outage until the caches around the Internet timed out. I probably could have avoided this if I contacted their support team prior to making the change, but I wasn't aware of this "gotcha!" But it has been smooth sailing since then.

Comments: