Today was a very discouraging day in some respects, and encouraging in others.. As one of the two "new" guys here on center the desire is to improve support and instill trust in the members here that the network will be reliable and online so that they can do their work and communicate with each other.
My first week here we were told there would be a power outage due to construction and that we should take advantage of that time to doe some maintenance.
We did a solid 9 hours of work, and the servers were looking better, although we were nervous that we didn't have more than 2 work days to plan for the large project (which normally took a month's time back in the corporate world). Both of us new guys, being detail oriented felt that there were a LOT of unknowns going into this project and we would try our best to get the work done.
After 9 hours, we were satisfied that it wasn't our best work, but it was the best we could do given the circumstances of our situation, namely, that we couldn't run to a store and pick up parts that we needed, nor did we have an extraordinary time to plan.
It was time to turn on the servers.
THIS is when things went bad. We knew, if things would fail, it would be now.
The disk array would not power on. This disk array held all of the exchange information (newsgroups, email, etc) and all of the corporate data.
To set the scene, CTS does not support workstation backups, everyone is encouraged to put their data onto the network device. So when it is down, they can't work. But it gets backed up twice nightly. Plus, the entire center's communication takes place via email and newsgroups. Hardly any events can happen without an announcement on the boards.
The short version of the story is that we couldn't fix the problem and there was no one around to help us, and so now, the network is down.
One thing I had not anticipated was the utter feeling of letting everyone down. At best all of my neighbors and everyone I serve, are inconvenienced but understanding, at worst, they are all angry and confused because there is no way to notify everyone since we don't have the knowledge on how to post on the web site the announcement.
So, everyone knows it's down, by now they know it's our doing, and they don't know when it'll come back online.
I have had outages before, some costing the customer millions of dollars, but I have never felt so closely attached to an outage as I do now. My first impression could not have been worse if I personally took a hammer to each person's modem. And yet, it isn't the dent in my reputation that stings so much as the fact that I have let down all my neighbors when I was trying to lift them up.
God is on the throne. He is in command. I will learn my lessons from this set back, and proceed forth with more humility than I had before.
Well, it is now Sunday night. The systems are up and running after a good effort this weekend and the fact that our manager returned and was very helpful. What I was not expecting was that the community here would not only be understanding but would be sympathetic that we had to work the whole weekend.
I have been told that people here to not easily get upset at things like this. I was very encouraged to know that outside my door weren't hundreds of neighbors wanting my head on a pike, but instead they were gracious. I have a lot learn about living in a community like this, but it definitely is not what I was expecting. Perhaps I'm gunshy. I am used to a manager coming down hard, asking a ton of questions, and customers being irate. Here, people seemed gracious, sympathetic, and apologetic that we had to work the weekend, and our manager prayed we would have peace and told us not to worry about the situation.
Wow.
Crashes happen, it's part of the job, it's how you handle yourself during them, and how you recover from them that define your capabilities in this job.
WE had no way to recover.... we had no information on how the backups worked, and we didn't have the passwords to the servers we needed.
Saturday we worked a 13 hour day, and gave up. Sunday a manager returned from out of town and helped us and we worked 4 hours on a sunday.
During the entire weekend the network was down.
Here is the amazing part.
-the first reaction of everyone was to pray. We prayed when the server crashed, we prayed each time we went to fix something. I've never worked in that environment before.
-I was expecting that everyone would be mad, but instead people came by with cookies, and after the ordeal was over, people were emailing us thanking us for working the weekend, apologetic that we had to work sunday and saturday, and thanking us for the hard work....
I am honestly stunned.
We are receiving praise for having worked through the problem, which could easily have been avoided. I'm not sure that everyone realizes it could have been avoided.
But I am simply amazed at the way people are reacting to this. There is a lot of grace here. Honestly it is not like working in any place I have ever worked before. I have yet to be chewed out, repromanded, or asked to determine root cause. I haven't been called into meetings on how this can be avoided in the future, and I don't have a swarm of managers hounding me.
In fact what I do have is people asking me what my favorite "treat" is to thank me, and my manager telling them it's "chocolate" (he doesn't know me well enough to know I am not the largest chocolate fan but still the sentiment is appreciated).
It makes me feel very loved, sort of how I feel when I consider that Christ loved us knowing full well our sin. The people here have welcomed us, not shamed us or belittled us.
THAT is perhaps the biggest culture shock I've felt yet.
Don't get me wrong, I'm fairly certain people are perturbed. You can't please everyone all the time, but I'm very delighted to see that the majority of responses have been more positive than negative, which, to a man used to the negativity, it is a delightful change of pace.
