12

RA Status: Update 6 (Read 324 times)

    Hi all,

    I managed to bring the disk arrays back online.  However, the contents were corrupt beyond repair.  I have initiated a restore of all file based data (gps workouts, maps, avatars, profile pictures, etc) that you've uploaded between June 16 and June 19.  Since all data files are backed up nightly, there is some permanent data loss.  If you have uploaded GPS workouts on June 19 between 4 AM EDT (8 AM GMT) and 7:30 PM EDT (9:30 PM GMT), you will need to re-upload those workouts if you want the maps data.

     

    After all backed up files are restored, I will begin the migration of all the data back onto the primary file server.  Due to the volume of data, the migration will take a couple of days.  Switching the web servers back to use the primary file server will not incur any down time, although the processing of your imported workouts might be delayed by an hour or so.

     

    I started looking into file system replications a couple of months ago.  Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill.  If you have worked with any DFS technologies, I like to hear your thoughts.

     

    eric Smile


    Fat & lazy runner

      Everything is ok for me, now!

      Thank you very much Eric!

       

      Bye

      Andy G.


      In it for the long run..

        The only run I don't have maps data for is June 20 (am).   I can certainly live without it.    Thanks!

        "It's not who wins the workout..."

        JML


          I started looking into file system replications a couple of months ago.  Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill.  If you have worked with any DFS technologies, I like to hear your thoughts.

           

          eric Smile

           

          I run technology for my company and we use a variety of replication technologies.  A decent one that can handle what I would guess is your file pattern (lots of smaller files) is DoubleTake from Vision Solutions.  It can also do automated failover for a number of server operating systems and we use it to handle replication and failover for a number of our services.  PM me if you want to chat further.

           

          Thanks

           

          Jon

           2014 goals: run a bunch....race some.....repeat...


          delicate flower

            I also checked my June 20 map data and I get a fatal error.  I don't need it either but just wanted to let you know.  Thanks for all the hard work, Eric!

            proud sherpa


            On the road back

              although the processing of your imported workouts might be delayed by an hour or so.

               

              eric Smile

               

              Ok.  Just looking to clarify so that we don't have a repeat of the "multiple upload" fiasco that caused more grief for you last time.  I uploaded a workout this morning and got the message that I could keep browsing the site and a little bar would appear and let me know when it was processed.

               

              My questions:

              1. My runs usually process immediately, so is this the delay you're talking about?

              2. When the status bar has appeared, it's usually appeared by the time I click through to another page, but I still haven't seen it and it's been a few minutes.  Is this to be expected?

               

              I'm cool.  No worries.  Just don't want to try and re-upload something that can add to your problems.  Thanks as always for all you do!

              2013 Goals: Run more. Smile while I do it. Complete 2 marathons in a year. (1 down, 1 to go)

              BJ Moose


              Running Over the Hill

                I started looking into file system replications a couple of months ago.  Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill.  If you have worked with any DFS technologies, I like to hear your thoughts.

                 

                eric Smile

                 

                If I'm understanding your needs correctly, another candidate (in addition to JML's suggestion) would be WAFS from Globalscape. Flexible, does what it's supposed to, and does not use not too much overhead.

                 

                BJ

                  Right now, all of the files are still stored on the backup file server in the second data center.  The two data centers supposed to have a 30 Gbps fiber optic link between them.  I have noticed a pattern the last couple of days where the GPS importer's queue is backed up, that users are uploading files faster than the importer can process.

                   

                  It's not that there are many files.  I think the problem is with the high ping time of about 90 ms.  Every file has hundreds or thousands of data points, and each point's elevation needs to be replaced.  The latency can really slow down the import process.  I have sent an email to the data center's support personnel.  Let's see what they have to say about it.

                   

                  As I noted in my original post, I am in the process of copying the data back onto the primary file server.  I've copied 74% of the elevation data.  The remaining 26% will take about 5.5 hours.  Once the elevation data is copied back, the importer should plow through all the queued workouts within an hour, assuming the data center cannot resolve their performance issue before then.

                    The only run I don't have maps data for is June 20 (am).   I can certainly live without it.    Thanks!

                     

                    The disk array corruption was detected on June 20.  Once I diagnosed the problem, I brought up a temporary file server to house the imported data.  Shortly after that, the backup server came online so I switched over to use it.  Some of the imported workouts are still on the temporary file server.  I'll be merging that data back onto the backup server.

                      The only run I don't have maps data for is June 20 (am).   I can certainly live without it.    Thanks!

                       

                      Karen,

                      I don't know at what time you uploaded the June 20 workout.  It is not part of the data on the temporary file server.  I'm guessing that you uploaded it right before the file server had its fatal failure.  I'm sorry to say that I can't recover it.  However, if you still have a copy of it on your GPS, you can re-import it.  Be sure to delete the original one to prevent duplication.

                       

                      eric Smile

                         

                        I started looking into file system replications a couple of months ago.  Ceph looks like a good candidate, although its requirements are a bit high and might be a bit of an overkill.  If you have worked with any DFS technologies, I like to hear your thoughts.

                         

                        Have you tried DFS included with Windows Server 2008/2012?  I hear good things about it, not used it here though (we are not a Windows shop).

                        Do what you want, just how you like. Nobody has to know.

                          Not sure what you did, but RA is working for me and I can now run three times as afar as I could before. It's a miracle.

                           

                          Seriously, thanks for all you do Eric.


                          Feeling the growl again

                            My avatar was there before the issues, and disappeared along with a lot of people's.  Theirs came back and mine did not, so I checked and there is no avatar file saved on my account anymore.  This is not a problem, I will simply replace it, but since I had not seen this mentioned I wanted you to know.

                            "If you want to be a bad a$s, then do what a bad a$s does.  There's your pep talk for today.  Go Run." -- Slo_Hand

                             

                              My avatar is gone, too.

                                My avatar seems to have reverted to one from several years ago.

                                12