It's been a tough weekend for me work-wise.
I am a very visible person in our community when it comes to technical issues. If something fails, usually my name comes up. As a result I have learned to take some of the heat when things fail that are beyond my control.
Like today. Monday. Or rather the entire weekend.
Geeks will understand this, the rest of you may not.
We were in the middle of a dns zone transfer moving from one service to the next, when our registrar for this entire country went down. Within 24 hours all dns cache was gone and no one in the world knows about our domain now.
Email is failing miserably to us right now... and not queuing up anywhere because there is no existance of our backup MX records.
Suddenly and overnight, we ceased to exist in the DNS world.
It gets worse.
I can't reach our registrar. All their phone lines are busy.
I get a notice from someone, that a main telephone switch is down.
Wait.. it gets worse...
In the midst of hoping somehow the registrar knows what is going on I get another notice...
There is a huge DOS attack on the countries major ISP which is causing most of the international traffic to die.
So even if DNS were repaired and working, the rest of the world wouldn't be able to get the info.
WOW.
Sometimes and suddenly you're reminded you live in a third world country.
For those trying to email us and getting "denied" please be patient... and pray for technical healing for this issue.
The end result is that many people here are seeing email not working and wondering why. Only a few small are fully able to understand the depth of the outage and how absolutely helpless we are to fix it right now.
The humility God is teaching me is that while I was once in complete control of my own DNS, I had built large DNS server structures with failovers and backups... here I sit at the mercy of a technology I once had total control over. (well if anyone can have total control over this beast). It is ironic. I'm dying to fix it myself I simply don't have the access to the right servers at our ISP.
Thanks for listening, but also thanks for praying.
It's another reminder to me of why I'm needed here. It's good to be needed, although I'd prefer it if things never broke so that the work here could go on uninterrupted.
It is hard to be patient and wait for a solution when you know several hundred people are depending on you to fix it.
"Hey there's Chad, what's he doing eating lunch? My email is still broken!"
Though for all of this imagined hostility (my brain works overtime, and I take my responsibility very seriously), this community has actually been very gracious and forgiving and understanding. Sometimes we even get cookies. The other day someone baked us a cake as a thank you. I have been time and time again surprised and blessed by how this community reacts when things break for a while and then are repaired.
One story, a mom was sitting with her daughter complaining about email not working the other day and the daughter said "hey mom, Chad worked until 4am last night repairing the email, on a Saturday, so cut him some slack." WOW!!!! I was pretty impressed to hear that.
Living here is an interesting experiment in loyalties. People feel the pressure of email not working, and yet know me and are sympathetic that I'm working hard. I recently had my vehicle in for repairs and they took a very long time, but my good friend was working on it. I was torn between the American desire to bark at management because things were going so slow, and realizing that management was a good friend who I knew to be overworked. So what do you do? You share the pain and try to recognize that everyone is doing the best they can. We don't have the luxury of being disconnected with the people we depend on and therefore shouldn't indulge in pressuring people to serve us more quickly.
Note to all of you IT professionals. This place is a wonderful place to work. What it lacks in the "latest and greatest" in technological toys, it makes up for in having true satisfaction that your job has value, and that people appreciate you.
