Forums >News and Announcements>RA Status: Update 6
Hi all,
I managed to bring the disk arrays back online. However, the contents were corrupt beyond repair. I have initiated a restore of all file based data (gps workouts, maps, avatars, profile pictures, etc) that you've uploaded between June 16 and June 19. Since all data files are backed up nightly, there is some permanent data loss. If you have uploaded GPS workouts on June 19 between 4 AM EDT (8 AM GMT) and 7:30 PM EDT (9:30 PM GMT), you will need to re-upload those workouts if you want the maps data.
After all backed up files are restored, I will begin the migration of all the data back onto the primary file server. Due to the volume of data, the migration will take a couple of days. Switching the web servers back to use the primary file server will not incur any down time, although the processing of your imported workouts might be delayed by an hour or so.
I started looking into file system replications a couple of months ago. Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill. If you have worked with any DFS technologies, I like to hear your thoughts.
eric
Fat & lazy runner
Everything is ok for me, now!
Thank you very much Eric!
Bye
Andy G.
The only run I don't have maps data for is June 20 (am). I can certainly live without it. Thanks!
Out there running since dinosaurs roamed the earth
I started looking into file system replications a couple of months ago. Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill. If you have worked with any DFS technologies, I like to hear your thoughts. eric
I run technology for my company and we use a variety of replication technologies. A decent one that can handle what I would guess is your file pattern (lots of smaller files) is DoubleTake from Vision Solutions. It can also do automated failover for a number of server operating systems and we use it to handle replication and failover for a number of our services. PM me if you want to chat further.
Thanks
Jon
Rebuilding my aerobic base....racing next year.....nothing to see here....move along now.
delicate flower
I also checked my June 20 map data and I get a fatal error. I don't need it either but just wanted to let you know. Thanks for all the hard work, Eric!
<3
On the road again...
although the processing of your imported workouts might be delayed by an hour or so. eric
although the processing of your imported workouts might be delayed by an hour or so.
Ok. Just looking to clarify so that we don't have a repeat of the "multiple upload" fiasco that caused more grief for you last time. I uploaded a workout this morning and got the message that I could keep browsing the site and a little bar would appear and let me know when it was processed.
My questions:
1. My runs usually process immediately, so is this the delay you're talking about?
2. When the status bar has appeared, it's usually appeared by the time I click through to another page, but I still haven't seen it and it's been a few minutes. Is this to be expected?
I'm cool. No worries. Just don't want to try and re-upload something that can add to your problems. Thanks as always for all you do!
I write. I read. I run. One time, I ran a lot on my 50th birthday.
Paul
Running Over the Hill
If I'm understanding your needs correctly, another candidate (in addition to JML's suggestion) would be WAFS from Globalscape. Flexible, does what it's supposed to, and does not use not too much overhead.
BJ
Right now, all of the files are still stored on the backup file server in the second data center. The two data centers supposed to have a 30 Gbps fiber optic link between them. I have noticed a pattern the last couple of days where the GPS importer's queue is backed up, that users are uploading files faster than the importer can process.
It's not that there are many files. I think the problem is with the high ping time of about 90 ms. Every file has hundreds or thousands of data points, and each point's elevation needs to be replaced. The latency can really slow down the import process. I have sent an email to the data center's support personnel. Let's see what they have to say about it.
As I noted in my original post, I am in the process of copying the data back onto the primary file server. I've copied 74% of the elevation data. The remaining 26% will take about 5.5 hours. Once the elevation data is copied back, the importer should plow through all the queued workouts within an hour, assuming the data center cannot resolve their performance issue before then.
The disk array corruption was detected on June 20. Once I diagnosed the problem, I brought up a temporary file server to house the imported data. Shortly after that, the backup server came online so I switched over to use it. Some of the imported workouts are still on the temporary file server. I'll be merging that data back onto the backup server.
Karen,
I don't know at what time you uploaded the June 20 workout. It is not part of the data on the temporary file server. I'm guessing that you uploaded it right before the file server had its fatal failure. I'm sorry to say that I can't recover it. However, if you still have a copy of it on your GPS, you can re-import it. Be sure to delete the original one to prevent duplication.
Have you tried DFS included with Windows Server 2008/2012? I hear good things about it, not used it here though (we are not a Windows shop).
Do what you want, just how you like. Nobody has to know.
Not sure what you did, but RA is working for me and I can now run three times as afar as I could before. It's a miracle.
Seriously, thanks for all you do Eric.
Feeling the growl again
My avatar was there before the issues, and disappeared along with a lot of people's. Theirs came back and mine did not, so I checked and there is no avatar file saved on my account anymore. This is not a problem, I will simply replace it, but since I had not seen this mentioned I wanted you to know.
"If you want to be a bad a$s, then do what a bad a$s does. There's your pep talk for today. Go Run." -- Slo_Hand
I am spaniel - Crusher of Treadmills
One day at a time
My avatar is gone, too.
My avatar seems to have reverted to one from several years ago.