DNS Pool Fault

Historical archive of system events. These events are posted for reference purposes only.
Locked
Seth
Site Admin
Posts: 295
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada
Contact:

DNS Pool Fault

Post by Seth » Sat Oct 03, 2009 7:55 pm

We experienced an general failure of our DNS caching pool for unknown reasons. We are looking in to it.
Last edited by Seth on Sun Oct 04, 2009 1:21 pm, edited 1 time in total.
Seth Mattinen, Roller Network LLC

Seth
Site Admin
Posts: 295
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada
Contact:

Post by Seth » Sat Oct 03, 2009 8:01 pm

It looks like the PowerDNS Recursor instances died.
Seth Mattinen, Roller Network LLC

Seth
Site Admin
Posts: 295
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada
Contact:

Post by Seth » Sat Oct 03, 2009 9:53 pm

We traced the problem to an APC dual-cord redundant switch that has - apparently - quietly failed. This caused a power loss to some equipment (namely the DNS pool and some servers with single power supplies) and affected other things.

This is the second failure we've had with these things and it'll be the last. I've removed the failed one and we will no longer be using these products. I'm going to crack open the case and see what I can find.
Seth Mattinen, Roller Network LLC

Seth
Site Admin
Posts: 295
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada
Contact:

Post by Seth » Sat Oct 03, 2009 10:15 pm

I broke open the relays inside and one of them had badly pitted and discolored contacts and the distinctive small of burned electrical equipment. I'll write up a newsfeed post with some pictures later.
Seth Mattinen, Roller Network LLC

Seth
Site Admin
Posts: 295
Joined: Sun Aug 30, 2009 10:44 pm
Location: Nevada
Contact:

Post by Seth » Sun Oct 04, 2009 10:13 am

Seth Mattinen, Roller Network LLC

Locked

Who is online

Users browsing this forum: No registered users and 1 guest