Apparently $DAY_JOB lost an airconditioning unit in the server room this weekend. And nobody got paged.
It then got very hot in the machine room. And nobody got paged.
The Emergency Power Off (EPO) triggered shortly before the sprinklers (and halon?) would have gone off. And nobody got paged.
After discovering the problem, several sysadmins spent all of Labor Day trying to get things back in shape.. but some hardware was apparently fried. When we got to work this morning, the ETA was noon. By noon it was 5pm. They sent us home. There seems to have been a little progress in the past five hours, but things are still not working properly.
The current rumor is that the paging modem was unplugged. It's almost funny how one $100 piece of equipment can cause problems in a multi-million dollar machine room.
For lack of a nail a shoe was lost...
