This morning I woke up, readied for church, and checked my email. Our #1 main server was down. This server is a dell 1950 with 2 dual core processors, running a virtual machine, our intranet, and our ftp server.
It has a powervault md on it, hosting 11 TB of data. That's 11 terabytes!
So to wake up to the thought of this server being down, meant a LOT of data loss.
Scenarios of disaster run through my head as I begin the trek to work. I start praying. "God please help me, I'm a month out of the seat, a little rusty, please guide me and heal this machine."
I went into work, and the disk controller was complaining of losing it's configuration.
GREAT... but .. um.. which controller is it? There are two, and if it's the big one, we've lost all data, if it's the small one, the server and all virtual machines, won't boot.
So...
A year and a half ago, we implemented a service called OCS, which inventories all our systems on the network. We very rarely need to refer to it, but this morning I did.
The database showed me quickly that the complaining controller was our "C" drive not our "D" drive and I was relieved.
Now... comes the scary process of pushing the right buttons.
When you're skirting the line of total data loss you hate to have warnings like "all disks are missing press C.. if you press C this action can not be reversed... are you SURE?"
and then "erasing configuration... this action will remove all data."
ugh... several scary steps later, I see the familiar "Windows 2003 server" splash screen and I stand up, triumphantly pumping both fists into the air..
"THANK YOU GOD!" I holler right as my co-worker walks in...
WOW... that was adrenaline. I went from being certain we had lost 11 terabytes of data to having not lost a single drop, and it only took 1.5 hours. Wow.
On top of that, I was able to discover root cause which is hard to do in these situations, and hopefully we can order spare parts to avoid such a scary scenario again.
Thank you GOD for answered prayer!!!
