September 11 Outage (Read 261 times)

Hi all,

RA suffered the longest outage in its history.  According to the log, connectivity was lost at 12:47 AM EDT.  Some of you emailed me soon after to report the problem.  Thank you.  I contacted the data center at 8 AM EDT.  They responded within minutes but it was from their support desk.  They forwarded the ticket to the technician on call.

After 6 hours since the ticket was created, and multiple emails requesting a diagnosis, the field tech finally told me that they are in the process of migrating all the servers' IP addresses to different routers and were encountering network issues since Friday.  He added that connectivity should be restored within minutes, but that didn't happen until a little over an hour later, at 3:20 PM EDT.  RA was inaccessible for 15 hours.

Since the problem was with the data center's routers, it meant the servers were operating normally.  They just weren't receiving any traffic.

While waiting to hear back from the data center, I was preparing to bring the backup servers online.  It is not a simple process, even though it should be.  All of the data were replicated to the backup servers so even if there's a catastrophic failure with the primary servers, there should be almost no data loss.  The backup server's web server was out of date so I had to update it before it can handle the traffic.  The problem was resolved before the web server was ready.

There are ways to reduce the total down time, even with a data center failure.  It requires significant infrastructure rearrangement.  I think the current setup should be adequate once the backup web servers are updated and ready to handle the traffic.  Using the current network layout scheme, the maximum down time can be reduced to one hour, which is the time it takes to update the DNS records to point to the new servers.

I have sent the data center another email requesting information about this outage.  If they respond, then I'll add the update here.  Thank you for your patience today.  It had been quite stressful.

eric

Thanks for keeping us up to date.
Your efforts are greately appreciated.

Eric - thank you. I cannot imagine how stressful this must have been as you take such pride in the customer service, design, functionality & reliability of RunningAhead.

I'm sure that everybody who uses this great site appreciates your hard work so once again, a big thank you from me!

Neil

UK

Thank you Eric.  We really do appreciate your hard work.

Thank you, Eric.  I only panicked a little!  I have faith in you!

Thanks for everything, Eric.

 

This reminded that I have been slacking and have not donated in a while.  Donation submitted.

Ditto- Thanks for all your hard work Eric!!  I think we should hold another pledge drive soon!

The data center manager replied to my displeased email but did not provide much additional info.  No surprise there.  Essentially, they added another router to the network to improve redundancy.  The new router and the old one were not communicating properly, resulting in some of the servers' traffic, including RA's, to be dropped.  There's really nothing anyone can do about that, aside from writing an angry email to help them remember this incident so that they'll do more testing during any future upgrades.

I re-upped my subscription just now. Sending you a big happy hug, if you're into that sorta thing. <3

Who doesn't enjoy a big happy hug?