12

RA Status: Update 7 (Read 606 times)

eric :)


    Hi all,

    Hopefully this will be the last update.  As of 7 AM EDT today, RA's primary file server finally came back online and is now serving the web servers.  It means the workout importer is no longer affected by the performance of the link to the backup file server and can process your GPS uploads nearly instantaneous again.

     

    As I indicated in a previous update, there was some data loss because file backups were done on a nightly basis.  GPS data uploaded on June 19 to June 20, depending on your time zone, cannot be recovered.  You'll need to upload them again.

     

    The data loss is the first in RA history.  I am looking for backup/replication solutions that will allow for more frequent backups without impacting the website's responsiveness.

     

    Thank you all for your patience and understanding through this ordeal.

     

    eric Smile

      And thank you for your monumental patience and diligence through this ordeal.

      Smile

      "I want you to pray as if everything depends on it, but I want you to prepare yourself as if everything depends on you."

      -- Dick LeBeau

      muppy


        And a big thank you Eric, you da man!

        Teresadfp


        One day at a time

          My avatar is back!  I missed the swan.  I have no idea where that photo is on my computer, so I couldn't get it back on my own.

            This was a tough few days for you.  Thanks so much for your diligence and hard work!

            Lou, (aka Mr. predawnrunner), MD, USA | Lou's Brews | lking@pobox.com

            HappyFeat


              Yes, thank you so much Eric for your diligence and expertise in resolving this huge problem!!!  We appreciate you sooooo much!!!!!

              Don't make excuses for why you can't get it done. 

              Focus on all the reasons why you must make it happen.


              running metalhead

                Great job mate!

                - Egmond ( 14 januari )            :  1:41:40 (21K)
                - Vondelparkloop ( 20 januari ) :  0:58.1 (10K but did 13.44!!!)
                - Twiskemolenloop ( 4 maart )  :   1:35:19 (3th M45!)

                - Ekiden Zwolle (10K)   ( 25 maart )
                - Rotterdam Marathon ( 8 april )
                - Leiden Marathon Halve ( 27 mei )
                - Marathon Amersfoort ( 10 juni)

                stadjak


                Interval Junkie --Nobby

                  The data loss is the first in RA history.  I am looking for backup/replication solutions that will allow for more frequent backups without impacting the website's responsiveness.

                   

                  All things considered, this is actually very impressive.  The data loss, being confined to a few days' logs, is so small.  Great job.

                   

                  I don't think anyone expects 5x9s from a free Running Log. Wink

                  2021 Goals: 50mpw 'cause there's nothing else to do

                  eric :)


                     

                    All things considered, this is actually very impressive.  The data loss, being confined to a few days' logs, is so small.  Great job.

                     

                    I don't think anyone expects 5x9s from a free Running Log. Wink

                     

                    Any data loss is unacceptable.  While nightly backups are standard practice in the IT world, the window is still too large given that users are uploading data 24 hours a day.  This episode exposed weaknesses in my backup procedures.  The recently added backup file server makes RA more resilient to equipment failures.  It also reduces the amount of down time in the event of a failure.

                     

                    On a related note, I am testing a backup solution that can make backups every 15 minutes without impacting performance.  There are solutions that can make nearly instantaneous backups.  I haven't ruled them out yet, although some of them have very high resource requirements while others seem to be no longer maintained.

                    LedLincoln


                    not bad for mile 25

                      Is a Google or Amazon data store a possibility?

                      eric :)


                        Is a Google or Amazon data store a possibility?

                         

                        I would love to move all my servers into the cloud.  The biggest benefit is I don't have to maintain the hardware.  I can pay as I go, and only pay for what I need.  I have to create servers that are more powerful than what I need at the moment so that they can handle future growth.  That money could have been saved using the cloud, but...

                         

                        The biggest drawback with the cloud is price.  I looked at Amazon's ECS, Microsoft's Azure, and Google's Compute Engine every so often, hoping that their pricing models have changed.  Their virtual machine options do not fit my usage.  It seems that the amount of memory for a given machine is proportional to the disk size.  I want a machine with lots of memory and a small disk as my database server.  The file server has the opposite requirement: large disks and (relative) little memory.  RA has about 1 TB worth of compressed data, all of which must be persistent.  Persistent storage is very expensive.

                         

                        In the end, owning the hardware and paying for space and bandwidth in a data center is still more cost effective than all other options I've explored.  It's much more work for me should something goes wrong, but it is what I can afford with the ad revenue and your financial support.

                          Thanks Eric. You rock.

                          Run, Walk, Crawl, just Finish.

                          HF # 1189

                          LedLincoln


                          not bad for mile 25

                             

                            I would love to move all my servers into the cloud.  The biggest benefit is I don't have to maintain the hardware.  I can pay as I go, and only pay for what I need.  I have to create servers that are more powerful than what I need at the moment so that they can handle future growth.  That money could have been saved using the cloud, but...

                             

                            The biggest drawback with the cloud is price.  I looked at Amazon's ECS, Microsoft's Azure, and Google's Compute Engine every so often, hoping that their pricing models have changed.  Their virtual machine options do not fit my usage.  It seems that the amount of memory for a given machine is proportional to the disk size.  I want a machine with lots of memory and a small disk as my database server.  The file server has the opposite requirement: large disks and (relative) little memory.  RA has about 1 TB worth of compressed data, all of which must be persistent.  Persistent storage is very expensive.

                             

                            In the end, owning the hardware and paying for space and bandwidth in a data center is still more cost effective than all other options I've explored.  It's much more work for me should something goes wrong, but it is what I can afford with the ad revenue and your financial support.

                             

                            Obviously, you've done your homework (I should have known).  Thanks again for all your dedication to the members of this site!


                            A Saucy Wench

                              Thanks for the explanation on the cloud eric.  Otherwise my dh would have worn your ear off about it .

                              I have become Death, the destroyer of electronic gadgets

                               

                              "When I got too tired to run anymore I just pretended I wasnt tired and kept running anyway" - dd, age 7

                                Thank you Eric!

                                12