I'm curious why this topic started. Just didn't seem obvious to me. So now, recently Google has been rummaging through my forums, as small as they are. I can read all the messages in one sitting and without a drink of water. BUT, Google seems to think it's interesting and I'll post the stats on that as output by AwStats below. Anyone else getting this “harsh"attack" from Google?
robot hits bandwidth lastvisit
------------------------------------------------------------
Googlebot 298469+38 5.79 GB 24 Aug 2005 - 22:42
Okay, so that is obviously this months Googlebot stats alone. Matter of fact, I'd have to check to be sure, but I'd say over ten days, fourteen tops. Next closest bot in bandwidth is only at 48.24MB.
I am well versed in robots.txt, not the point here.