{"id":832,"date":"2009-01-06T21:30:23","date_gmt":"2009-01-07T05:30:23","guid":{"rendered":"http:\/\/multimedia.cx\/eggs\/?p=832"},"modified":"2009-01-06T21:31:21","modified_gmt":"2009-01-07T05:31:21","slug":"this-is-what-i-was-trying-to-avoid","status":"publish","type":"post","link":"https:\/\/multimedia.cx\/eggs\/this-is-what-i-was-trying-to-avoid\/","title":{"rendered":"This Is What I Was Trying To Avoid"},"content":{"rendered":"

I checked my website bandwidth overview tonight. So far for the month of January, the bandwidth served from the main multimedia.cx domain is actually much higher than the bandwidth served by my gaming blog<\/a>, which never happens (lots more pictures over there). I dug a little deeper into the details and found this:<\/p>\n


\n\"Googlebot
\n<\/center><\/p>\n

So who is 66.249.67.1? Why, none other than crawl-66-249-67-1.googlebot.com. Why has it taken such an interest in my site? Oh, little pages like this:<\/p>\n

66.249.67.1 – – [01\/Jan\/2009:00:58:01 -0500] “GET \/fate\/index.php?stderr=41851 HTTP\/1.1” 200 69107 “-” “Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)”<\/p>\n

66.249.67.1 – – [01\/Jan\/2009:00:58:06 -0500] “GET \/fate\/index.php?build_record=43652 HTTP\/1.1” 200 3297 “-” “Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)”<\/p>\n

You see, I thought I had administered my FATE web database<\/a> responsibly by adding the appropriate robots exclusion file<\/a> at http:\/\/fate.multimedia.cx\/robots.txt<\/a> by simply disallowing crawlers at this point. I completely neglected that http:\/\/multimedia.cx\/fate\/<\/a> is a perfectly valid route into the site.<\/p>\n

Lesson learned<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

I learned the hard way that web sites have multiple routes of entry<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[101],"tags":[],"class_list":["post-832","post","type-post","status-publish","format-standard","hentry","category-fate-server"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t