Today, Friday, was a normal day. Do a few network patches at the school, double check backups, tweak a few servers.
But then at 3pm, the power went out.
Our server room is powered by a room full of batteries connected to an inverter. We used to use Deep Charge but they were hard to come by.
ANYWAY....
when the power goes out, the servers run on battery power. BUT, the air conditioning doesn't run.
Very quickly our little room of servers begins to heat up, and when that happens the hardware begins to fail.
Typically turning on the generated is automatic after 5 minutes.
So the power goes out..
we wait 5 minutes... no generator
we wait 10..
we keep our eye on the thermometer readout...
it's growing.
We call the electricians "hey what's going on, we're going to have to start shutting down."
today the answer came (welcome to life here) "We can't get a hold of [the guy who knows how to start the generator]"
My guess was he was watching his kids compete at the annual sport's day...
So I ask a co-worker with a radio, "can you call him on the radio?"
"he doesn't have it on him."
So.. a guy nearby him with a radio hears the call, taps his shoulder.. OKAY now he heads to the generator room.
Uh oh.. problem in the fuel line... he begins to work.
The call comes back "20 minutes to an hour before it's up"
So.. we... the network guys, at 3:15pm begin shutting down servers, which at this point, are basically heaters with strong fans.
Services go down, but people don't mind because they're houses are out of power too.
We finally get down to the bare minimum... open all the windows and doors, and the servers stabilize and begin to drop in temperature.
For those that are aware of the "watchdog reset"... ours kick off around 31 degrees celcius. The inside of the servers got to 29 degrees celcius.
That's 2 degrees... PHEW.. we made it.
If we let them get too hot... parts melt and break and it costs us money.
So it's not frantic but it is a little bit of... hustle.. bringing things down so they don't break.
Well it's all back up now. This happens a few times a year, and when it does... the computer/network geek appreciates three things in a major way:
-Air conditioning - and those who can maintain it
-Electricity - and those who can maintain it
-CB/HF radio - and those who can operate them
Keeping this room running safely was truly a team effort, and if there is one great thing to say about this place, it's that people pull together during a crisis, be it ever so minor.
