Security Upgrades (Read 277 times)

Hi all,

The latest spade of bugs were caused by the security upgrades that I put in this weekend. The SSL certificate protecting all traffic coming into and out of RA was about to expire.  The project started with installing a new one.  That was straight forward enough.

RA has several web servers.  When you go to a particular page, your request goes to a load balancer, which forwards your request to one of the web server.  If one of the web server is offline, which happens every few days due to security patches and RA updates, the load balancer would reroute traffic to other web servers.  In theory, you would never know if a web server is offline because the load balancer checks the status of each web server constantly.  In practice, there is a few seconds delay between the web server going offline and rerouting the traffic, because there is no programmatic way to tell the load balance that a web server is going offline.  The end result is you'll get an error page instead of the page you requested.

The next task was to switch RA to use a different load balancer, one that the deployment system can notify when a web server is about to be taken offline for an update.  The new load balancer is working as expected.  I've made multiple updates to RA since and I don't think anyone has received an error page.

To make sure the load balancer is configured correctly, and everything is updated, I ran a test using Qualy's SSL Labs Tools. I'm no security expert and I do no pretend to be one.  I do know that security is hard and it's best left to the experts.  The tool scans RA's web servers to ensure there are no known exploitable security holes.  I run the scan every few months.  I was expecting an A grade as usual but received a B.  It turned out that the load balancer was still allowing a weaker cipher.  After I fixed it and making other enhancements, RA received an A+.

We are all trained to look for the security lock icon when we want secure web browsing.  Many if not most people don't know that the lock is overly simplistic.  It just means the page uses some kind of encryption.  It doesn't tell you the strength of the encryption, or other security best practices.  Having the lock icon doesn't mean the website is safe.  It could be using a weak encryption algorithm, one that can be easily cracked.

A website's security measures weaken over time as computers get more powerful and vulnerabilities within encryption libraries are discovered.  An example of a potential threat is an attacker can capture and store all traffic between you and the websites that you visit, and wait until a time when computers are powerful enough to decrypt the data using brute force methods.  Very few hackers have the resources and capabilities to do this, except for government spy agencies such as NSA and Russia.  I doubt any government is interested in your data, but you may feel differently.  Therefore, RA's servers employ something called forward secrecy.  What that means is that the encryption key used to protect the connection between your computer and RA's servers is changed frequently.  Even if the hacker somehow finds a key to decrypt some of your page views, the same key cannot be used to decrypt all your page views.  Currently, finding a single key using brute force takes many years.

In order to get the A+ grade, RA has to implement HSTS, which stands for HTTP Strict Transport Security. That's just a nerd way of saying that RA requires all traffic to be encrypted, and don't let anyone tell you otherwise.  When you're traveling, you'll probably stay at a hotel.  All hotels offer WiFi access, and they're notoriously insecure, because security is hard.  A hacker in another room can intercept your data and trick your computer into not using encryption.  If you have visited RA before the trip, your browser will refuse to access RA without using encryption.  More interestingly, even if I disable SSL encryption on RA today, your computer will still refuse to talk to it.

The switch to use HSTS broke parts of RA.  There are a few old links to JavaScript files that were hard coded to use HTTP (insecure) connection.  Since RA told your browser to only talk to RA securely, your browser won't download those files, thus breaking any page that contained those links.

The switch also broke the Garmin Communicator.  In order to use the plugin, I have to supply the website's URL and an access key, which is specific to each website.  I don't know why the URL is needed since the Communicator plugin checks it against what it thinks the correct URL should be.  Additionally, in Garmin's brilliance, the key is also sensitive to the exact URL.  That is, it thinks http://www.runningahead.com/ is not the same website as https://www.runningahead.com/.  The only difference is the s in https.  Technically, they are different, but I don't think anyone would think they point to different websites.  Since the JavaScript file was still passing the insecure URL to the plugin, the plugin refused to be unlocked.  The fix protected the correct URL to the plugin, thus unlocking it.

The last bug was despite the fix, the Communicator plugin still can't be unlocked.  That was because all JavaScript files have versions and are cached on your computer to increase performance.  The Communicator JavaScript file's version wasn't updated with the fix so your browser didn't download the latest one.  A version update fixed that problem.

I think the system is stable again.  Sorry for the techno babble.  I tried to simplify the details but not water it down.  Let me know if you find any problems, or have any concerns.

eric

Thank you, Eric - for your work and the explanation.

(I might use part of it when I have to explain to my mother - again - how online "security" measures may be a PITA but shouldn't be disabled/bypassed for momentary convenience [as a hapless user])

No idea what you said, thanks for the hard works regardless. But maybe this has something to do with why the ancient HandyRunner app isn't uploading runs to the site directly anymore? If there's an easy fix, it would be appreciated.

Thank you for all your work! Security is hard, indeed, but you are taking all the right steps to ensure we are protected to the best of our abilities. I am thoroughly impressed with the dedication to security by a running website

Garmin is driving me nuts. Maybe they want me to upgrade from my old Forerunner 405. Anyway, I finally got it to add my run to RA.

 

Maybe I should just use my iPhone RunForth app. I don't know if there is a similar app for Android.

You need to get the Forerunner 42 (See what I did there?)

The first time I've heard about it. Thanks for the info.
I do not do much in the other direction. If you are interested, you can look at it here.