Mock me if you must, but since high school I have believed in the power of prayer on hardware. Since that very first time I was troubleshooting a broken professional tv Video Tape Recorder. The teacher said 'Chad this isn't working see what you can do.' I spent an hour on it, couldn't resolve it, put my hands on it and prayed. It started working.
We don't battle against flesh and blood alone, but in my job it seems that is my primary function. To battle against the physical and logical failures of computers and networks.
Today we had the most catastrophic failure here that I've ever seen.
-one of our VM servers had a hardware error which crashed 8 virtual machines
-when the second VM server came up, and brought them back up, they were in a confused state because the boot sequence was broken (our Active directory came up last)
-at the same time, a motherboard lost a sensor and refused to boot on our main backup server... which means, not only would we not get a full backup tonight, but backups last night probably failed.
-all of this, on top of massive email failures, because one of our smarthosts went down and was refusing connections and I had 900 email messages in the queue.
In short, almost everything was broken at the same time, and I could not find a common cause. Typically you could look at power, but that wouldn't explain why the datastore filled up.
So as we sat there, working, trying to get it back online, I got to a point where I said 'wow! This is a lot to fix.' The phones were ringing, I was both network manager and acting department manager.
I was stuck. I had to fix these things with my co-worker, and had to do it fast.
I stopped and looked, and closed my eyes and prayed.
I put out a quick shout to some of the people calling in to tell me what I already knew (that it was broken)... and said 'all you can do is be patient, and pray'.
"God, if this is some random act of failure, please give me the ability to solve it quickly. If this is a spiritual thing, I pray against the enemy and his attempts. You are greater than he, and I pray you would bring healing to this equipment, bring it online, and I thank you for the skill you've taught me over the years, please make my brain work well right now."
It wasn't even 2 minutes before things starting coming online again.
I can't explain to you the joy that comes from seeing all reds, to all greens, in a very short amount of time.
In the end, we still haven't been able to diagnose a single root cause. It seems like multiple simultaneous failures.
We lost 1 server, the motherboard is dead.
But, as God would have it, we had another server recently offlined (we virtualized it), so we could steal it's motherboard.
We brought the crashed backup server back online in 2 minutes or so, and guess what.
The crashes happened in such a way last night, that we got all of our backups!
Not only did stuff come back online, but when it went OFFLINE, it went in such a way that we still did not lose a single bit of translation data, and all the other data that enables this work. And email is routing properly now too!
Some may cry 'coincidence'. Others may say 'well that's why you have I.T. guys'.
I say, God is great. He spared us.
It could have been really really bad, and I think, it would have been if we didn't realize we needed to pray. If we didn't submit and try to continue on in our own way, sure it may have gotten fixed eventually but God's way is so much better.
It is SO awesome that we're back online, what a crazy morning, but it ended without sorrow and that's a good thing.
