We traced the problem to an APC dual-cord redundant switch that has - apparently - quietly failed. This caused a power loss to some equipment (namely the DNS pool and some servers with single power supplies) and affected other things.
This is the second failure we've had with these things and it'll be the last. I've removed the failed one and we will no longer be using these products. I'm going to crack open the case and see what I can find.
Seth Mattinen, Roller Network LLC