Unavoidable bugs (Read 1431 times)

When it comes to software engineering, bugs are expected.  We don't purposefully write bad code, although some programmers might if they're disgruntled.  Most of us take pride in our work.

Most bugs could be avoided simply by thinking through the logics.  Others are harder to find because they involve interactions amongst multiple components.  They only happen under very specific conditions and are nightmares to replicate.  Still, some are completely beyond a programmer's control.

Whenever RA encounters an unexpected error, it sends me an email with details about the problem to help me fix it.  I get more of these emails after a site update.

RA was unavailable for many users yesterday morning.  I received tens of thousands of emails from RA describing the same problem: "There is already an open DataReader associated with this Connection which must be closed first."  You don't have to understand what it means other than that was the error I received and it was related to the database.  This error is considered catastrophic because it prevented users from using the site.  Is is almost as bad is the site being completely down.

I encountered this problem years ago and only during development.  I made the necessary changes and it never happened again.  I was quite perplexed when I saw this error yesterday because I didn't think it was possible anymore.  Since the computer never lies, the problem was there and finding it would be like finding a needle in a haystack.

I examined all the changes I made in the past several days, thinking one of them may have caused it since the problem only popped up yesterday.  Everything looked fine.  I looked at all the changes in specific components that are part of the latest update and that came up empty as well.

As a last resort, I went through all the source code that interacted with the database but didn't find anything obvious.  Without a smoking gun, the only thing I could do was to add a bit of code that guarantees this problem won't happen ever again.

I rolled out the fix last night when the site was the least busy and went back to working on other stuff.  Several hours later, RA sent me emails with the same problem again.  The only way to fix it was to restart the web server.  I can't monitor it constantly.  Crap!

I looked at server logs, database logs, performance monitors and whatever else I can think of but nothing obvious.  I looked at many of the emails hoping to find a pattern but nothing incriminating.  Half an hour later, another torrent of emails arrived, signaling the return of the problem.  I restarted the web server and resumed my investigation.

In both batches of emails, the majority of the page requests were by a web crawler called Baiduspider+.  It is a Chinese search engine.  What's more, this crawler requested many pages per second.  The first batch of emails occurred just as the nightly backup completed.  It is highly possible that this problem occurs only when the server is at a high load.

I overloaded my test environment to replicate the scenario but was not able to reproduce the problem.  Still, I know this has to be the cause.  Out of desperation, I Googled the problem.  After several tries with different keywords, I finally found my answer.

The latest RA release contains an updated MySQL database access library, ironically to fix another bug.  Someone reported a critical bug against the access library that occurs when the database is under high load with the same error message.  Bingo!  There is a fix but it is not released for public consumption yet.  Since this fix is vital to RA, I had to download the latest source code and build it myself.  I will upload the fix tonight when the server is less busy.

As a whole, I am happy to know that I was not responsible for the bug despite spending over a day tracking down the problem.  It goes to show that even if you write perfect bug-free code, bugs are still inevitable.

Thanks for all the work, eric with the sideways smile.

'Since the computer never lies'

I know I've made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal. I've still got the greatest enthusiasm and confidence in the mission. And I want to help you.

Thank you!  Thank you! 

I don't have the faintest idea what all that computer talk means, but it makes my head hurt in sympathy. Thanks for the sleuthing.

Oh.  I thought this was another insect encounters thread.



GREAT JOB RA-KING!

Huh, I was trying to type Eric with the sideways smile, but the system converted it to a bright yellow happy face.  Oh well.

I thought this said unavoidable HUGS. 

 

hehe.