RunningAHEAD - Topic: RunningAHEAD status

eric :)

posted: 3/7/2020 at 12:35 AM

Hi all,

tl;dr: I'm not expecting any data loss but some data will take a while to be restored.

RunningAHEAD.com's production servers suffered a fatal failure some time today. I am still unable to connect to the servers so I'm assuming the disk array that they use have completely failed. I have been trying to switch over to the backup servers this past week but encountered unexpected problems. When the data center moved the servers, the database server did not come up correctly. As a result, replication did not happen. By the time I noticed it, the production and backup databases have drifted too far apart for the replication process to resume. It took a couple of days to restore the backup database and resynchronize the databases.

The next problem was the file server. Its replication also failed. It was caused by the disk running out of space. The filesystem does not handle out of disk space well, and I think all the data on the disk was lost. I tried to restore the data from my secondary backup server but it failed after a few days due to the large volume of data involved. I tried to replicate the data from the production file server and I think the additional load on the disks was what ultimately caused the fatal failure.

Right now, the web server is running. The database is running and should contain the latest data. You should see all your workouts. The file server is up, but does not contain any data at the moment. I am preparing to restore all the data from the secondary backup server again, using a different method. It'll probably take several days to complete so I'll let you know when it's completed. You can continue to record your workouts during this time.

RA won't be able to accept data from Garmin Connect at the moment. I still need to setup another server to handle the traffic but that should be pretty quick.

The good news is I think RA will be more stable. I apologize for any inconvenience and headaches this has caused. If you have any questions or concerns, please let me know.

eric Smile

milktruck

an amazing likeness

posted: 3/7/2020 at 12:45 AM
modified: 3/7/2020 at 10:03 AM

What a pain on a Friday night Eric...hang in there. Three items for your to-do:

1. The log charting tool is using the development license, not prod.

2. user account avatars images are missing...that's related to the file server, I suspect.

3. manual file uploads aren't getting processed, either, in addition to Garmin Connect integration.

_{Acceptable at a dance, invaluable in a shipwreck.}

stadjak

Interval Junkie --Nobby

posted: 3/7/2020 at 1:16 AM

I'm sure we're all patient. Thanks, Eric Smile , for hopping on it. Sorry the gremlins got to the server. We'll try not to feed them after midnight anymore.

2021 Goals: 50mpw 'cause there's nothing else to do

Fredford66

Waltons ThreadLord

posted: 3/7/2020 at 6:13 AM

Thank you, Eric.

5k 23:48.45 (3/22); 4M 31:26 (2/22); 5M 38:55 (11/23); 10k 49:24 (10/22); 
10M 1:29:33 (2/24); Half 1:48:32 (10/22); Marathon 4:29:58 (11/23)

Upcoming races: Clinton Country Run 15k, 4/27; Running is Back 10k, 5/12

Baboon

delicate flower

posted: 3/7/2020 at 7:56 AM

<3

MadisonMandy

Refurbished Hip

posted: 3/7/2020 at 8:17 AM

Thanks for everything you do, Eric!!!

Running is dumb.

DavePNW

posted: 3/7/2020 at 8:51 AM

Quote from stadjak on 3/7/2020 at 1:16 AM:

the gremlins got to the server.

This is the only thing I’ve understood so far.

Dave

stealth.rnr

She laughs at me......

posted: 3/7/2020 at 9:09 AM

This is a good time to consider the $10 a year subscription or donate to RA. Eric owns and runs this place and it's worth much more than $10 a year.

Thank you Eric for all that you do.

Tramps

posted: 3/7/2020 at 10:48 AM

How I spent yesterday.

Thanks for all your work on this, Eric!

Be safe. Be kind.

Jaren Sr.

posted: 3/7/2020 at 11:34 AM

Great service, been using it for years. I'd be happy to donate as well. Let me know what I need to do?

Jaren

Madison, WI

Cyberic

posted: 3/7/2020 at 11:36 AM

Thanks Eric. That is a lot of work.

CanadianMeg

#RunEveryDay

posted: 3/7/2020 at 12:25 PM
modified: 3/7/2020 at 12:26 PM

Quote from Jaren Sr. on 3/7/2020 at 11:34 AM:

Great service, been using it for years. I'd be happy to donate as well. Let me know what I need to do?

Jaren

Madison, WI

Go into options at the top right hand of your screen. It's above the RA search bar. On the next screen, one of the boxes is green and says "subscription plan." That's the place to go. Smile

Half Fanatic #9292.

Game Admin for RA Running Game 2023.

evanflein

posted: 3/7/2020 at 12:51 PM
modified: 3/7/2020 at 12:51 PM

Thanks, Eric. I wonder why Tramps' avatar is the only one that shows up?

ETA: oops, and mine?

eric :)

posted: 3/7/2020 at 1:04 PM

Update:

Garmin Connect workouts are being processed again. It may take a while for your workouts to show up.
The file server is still being restored. I don't have an ETA on it at the moment. It might take several days. Until then, all your workout details (e.g. graphs), avatars, and everything else that's file based won't be available.
Elevation data won't be available until the file server is restored since the data is also file based.

mymagicalpants

posted: 3/7/2020 at 3:08 PM

Thank you Eric! This is still the best running/workout log out there by far. You do a great job.

RunningAHEAD status (Read 430 times)