Well I suppose since I just shared a success I should share a failure.
For a long time in my career I have been able to say "I have never had a large data loss." I've always been extremely careful about backups and double backups. But mostly however, that was on unix systems where duplicating disks and running raid was my cup of tea.
This weekend our major corporate email database (microsoft exchange) was corrupted while one of my co-workers tried to restore an inbox. After 8 hours of work and database recovery, I was able to restore ALL previous email but 2 days worth.
Still, 2 days worth is serious data loss to me.
Fortunately, out of the 600 people on centre, only about 5 people thus far have actually experienced data loss because most folks were running in Cached exchange mode.
I have not experienced any key people... so the data loss isn't HUGE.. and so far nothing vital was lost. I've even had people say "hey I love it, now I have a scape goat.. I can say 'oh sorry I haven't worked on that, you must have emailed me during the 'blackout'"
Still though, this weekend was a time for me of pondering if it had been better I didn't even wake up this weekend. I know my value here, I know God put me here, but it doesn't mean everything I do will be a raging success.
This data loss, while minimal, was the largest data loss of my entire career.
I guess that means I've had a blessed career.
I think the sting is a little harder because my customers are also neighbors and if they feel an email loss, so do I.
Formerly if I had down time like this, it would have been costing us millions of dollars, and so working here for me has been less stressful than working for a big corporation.
I did learn this weekend though, that I need to constantly be vigilent.
The entire error happened because I was very very busy handling 3 emergencies and someone asked "Hey Chad should I try this." and I said flippantly "yes" without thinking through the consequences.
I think from now on, I'll be more realistic about my abilities and try to be more cautious. I think handling 3 emergencies isn't a time to be answering questions I'm not entirely listening to.
I praise God! During the outage I was praying "God please, let the only person affected by this be me, let me pay the cost for my mistake, not everyone else working here."
And while I was upset about the failure and loss, God brought this to mind:
-If I hadn't split the email servers to home/corporate 2 months ago, this loss would have affected more than twice the amount of people.
-Cached mode saved most emails for most people.
-You've learned a valuable lesson from this.
Yes I am humbled today. It is a good thing to be humbled from time to time. And I personally LOVE that God doesn't seem to let me get out of check. I constantly pray that God would not let me think that what is done here is of me, but instead of Him.
This weekend could have been HORRIBLE but instead it was only bad. That's a huge reminder that the skills I have are from God and my attitude should reflect that.
It was fun being a hero last week, but it was also enjoyable in a different way, to watch God be the hero this weekend.
