Massive panic at work today. It seems that the temperature controls failed in the main server room at one of our data centers and it got up to 106 F. So far it looks like the machines that we care about all shut themselves off before meltdown, but it still caused major problems and it will be a while before we know if there was any permanent damage.
Ouch.
I've had that happen a few times
hfb on 2003-04-16T14:51:21
the suns can take a baking but you're sure to lose a few PCs over the next few weeks so maybe order spare parts now :)
Definitely order spare parts...
Elian on 2003-04-16T16:04:23
I had this happen once as well, only I was "lucky" enough to have one of the server boxes directly in the vent backblast from the big iron box. It got... toasty.
There were component failures on and off over the next three weeks. Don't be surprised to find memory errors, chip failures, or mechanical components dying for the next month or so. It would be wise to take at least a twice-daily walk through the machine room listening for the lovely sounds of fans and/or disk drives starting to die.
Not a bad time to review the backup procedures, which of course you have. Right?
:)
even backups might not save you
bbcrack on 2003-04-17T11:15:24
Once one of my coworkers left the aircon off over the weekend. We come back on monday rthings are fine but then a disk fails ( Fujitsu, so not surprising). So when we come to replace that another one fails. Totally hosing the raid 5 partion with the exchange information store on it. So we go to restore from backup. Guess what esle had failed.
The £3000 LTO drive. Which for some reason we diddnt have a support contract for. So we had to buy another or wait 2 weeks for warranty repair. This was fun.
