I want to apologize for the lack of posting, and certainly being late with this great topic Mark Pilgrim had addressed this past Wednesday. I’ve been busy rebuilding a more usable Redland Baptist web site and it has consumed much of my free time. Don’t worry, I’m taking notes, and I intend to share them with the rest of you.
About a month ago, Vincent Flanders had emailed me. He was getting harvested hard by various spambots. I was reminded of that dialog when I read Mark Pilgrim’s 26-Feb-03 post entitled How to block spambots, ban spybots, and tell unwanted robots to go to hell. Here is what Mark said on his site — which sounded very similar to the original email I had received from Vincent:
I recently woke up to the staggering level of abuse occurring on this web site. This is old news to some, but we all wake up at different times. I’m talking, of course, about automated robots and spiders. They come at all hours, they take as much as they can, and they leave me with the (bandwidth) bill. They do so without respecting the Robot Exclusion Standard, now almost 10 years old. Some come to gather email addresses, which are then sold to spammers; some come to steal images or other content, and republish it without my consent; some come to spy on me and sell information to their clients about perceived violations of copyright, trademark, or some nebulous concept of brand identity.
None of them act in my best interest, or in the interests of my readers.
And this is why I’m adding Mr. Pilgrim’s to the mandatory reading list.