This Is What I Was Trying To Avoid

I checked my website bandwidth overview tonight. So far for the month of January, the bandwidth served from the main domain is actually much higher than the bandwidth served by my gaming blog, which never happens (lots more pictures over there). I dug a little deeper into the details and found this:

Googlebot causing bandwidth havoc

So who is Why, none other than Why has it taken such an interest in my site? Oh, little pages like this: – – [01/Jan/2009:00:58:01 -0500] “GET /fate/index.php?stderr=41851 HTTP/1.1” 200 69107 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +” – – [01/Jan/2009:00:58:06 -0500] “GET /fate/index.php?build_record=43652 HTTP/1.1” 200 3297 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +”

You see, I thought I had administered my FATE web database responsibly by adding the appropriate robots exclusion file at by simply disallowing crawlers at this point. I completely neglected that is a perfectly valid route into the site.

Lesson learned.

One thought on “This Is What I Was Trying To Avoid

Comments are closed.