The worst Internet Outage Ever.... is OVER.. we're back online!
Sunday night as we went to bed, my wife and I said 'this is going to be
a busy week'. It was one of those weeks when you pray no one gets sick
or injured because you can't flex enough to cover it.We had extra work
responsibilities, I had a scheduled network maintenance which would take
a half a day, departments were closing (which makes my life a bit harder
sometimes), it was the last week of school term, and thus concerts to
attend on weds and friday. My daughter's schedule is hectic, I'm
performing in a farewell concert this weekend, and generally life was so
booked up, that we could not possibly fit any more in.
We had to fit more in.
On Monday I packed up a lot of equipment and shipped it to a site I was
scheduled to do a remote VSAT installation. I got a call saying our
nephew was gravely ill (he no longer is) and the entire family was sent
into a tailspin with this horrible news. Tuesday we found out it was
bogus, and that was GREAT news. Phew, close call there, we nearly had
to start the equations of figuring out another trip home?!?!
Then weds the internet stopped working. I'm alone in the network room,
we're short staffed, and I can't figure out why, so I call the ISP.
Long story short, life got a whole lot busier as I started pulling a 20
hour day.
It turns out the equipment on our end failed, and we had no spares. It
took us an entire day to troubleshoot because we had no internet
connection, no way to talk to the outside world but by cell phone.
In the U.S. the worst outage I can recall was 8 hours long. We were
able to get hardware replacements VERY quickly. Here, the best time
looked like 3 days away. The internet has been so stable and reliable
for the past several years. In fact during my entire time here, it's
never been this bad. We're on day 3 as I right this.
Last night, there was the possibility of getting a replacement on a
plane, but the guys with the spares, missed the plane by 2 minutes. They
saw it taking off as they pulled into the airport. Around 8:30 I gave up
trying plans B and C. Long day started at 4:30am.
I was able to choke down a three minute lunch, attend most of my
daughter's band concert, and say hi to my son.
This morning, I got a call 'we're able to divert a plane to Madang'.
Okay, THAT is cool. The internet being down is a crisis but not a fatal
one. On my say, a Kodiak plane and it's pilot were diverted from their
schedule of 30 minutes away, to land at a different airport, and pick up
spare parts. Suddenly there is a hope that the spare parts will arrive
here in 2 hours from now!
This outage has been the worst in my career. It is something I have
tried to avoid and plan against, but we couldn't justify having $15,000
worth of spare equipment lying around. Until recently that wasn't an
issue in that we had VSAT installations sitting around waiting to be
shipped, and so would have had spares anyway. But on Monday I shipped
all of that to the final installation.
So my tools, cables, testers, all of it, gone, during what has become
the worst internet outage of my career, and of recent Ukarumpa history.
(I heard a tale that before when they were using leased lines they would
lose regularly for 3-4 days at a time and send email via a hard drive on
a plane to a place that had internet).
As any network admin knows, when your network is down, the up side is
that you can get a lot of maintenance done, which I did, taking the time
to clean up our horrible network cable mess.
Also thank you to a friend who donated my original iPhone to me years
back, I was able to do most of the network troubleshooting with an
iPhone because our cellular network was the only thing working at the
time. I've written Apple with a testimonial in hopes they might want to
send me an iPhone 4s for free (-;
