DNS Pool Fault

Historical archive of system events. These events are posted for reference purposes only.

DNS Pool Fault

Postby Seth » Sat Oct 03, 2009 7:55 pm

We experienced an general failure of our DNS caching pool for unknown reasons. We are looking in to it.
Last edited by Seth on Sun Oct 04, 2009 1:21 pm, edited 1 time in total.
Seth Mattinen, Roller Network LLC
Seth
Site Admin
 
Posts: 280
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada

Postby Seth » Sat Oct 03, 2009 8:01 pm

It looks like the PowerDNS Recursor instances died.
Seth Mattinen, Roller Network LLC
Seth
Site Admin
 
Posts: 280
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada

Postby Seth » Sat Oct 03, 2009 9:53 pm

We traced the problem to an APC dual-cord redundant switch that has - apparently - quietly failed. This caused a power loss to some equipment (namely the DNS pool and some servers with single power supplies) and affected other things.

This is the second failure we've had with these things and it'll be the last. I've removed the failed one and we will no longer be using these products. I'm going to crack open the case and see what I can find.
Seth Mattinen, Roller Network LLC
Seth
Site Admin
 
Posts: 280
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada

Postby Seth » Sat Oct 03, 2009 10:15 pm

I broke open the relays inside and one of them had badly pitted and discolored contacts and the distinctive small of burned electrical equipment. I'll write up a newsfeed post with some pictures later.
Seth Mattinen, Roller Network LLC
Seth
Site Admin
 
Posts: 280
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada

Postby Seth » Sun Oct 04, 2009 10:13 am

Seth Mattinen, Roller Network LLC
Seth
Site Admin
 
Posts: 280
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada


Return to System Status Archives

Who is online

Users browsing this forum: No registered users and 1 guest

cron