So, do I be a bastard and ignore them? Or do I unblock them?
Argument for ignore them: Shitloads of robots blatantly ignore robots.txt and cause vast amounts of trouble (usually wasted CPU and network resources). Ask in particular got stuck in a spammer tarpit that generates infinite numbers of bogus pages with bogus email addresses on them designed to clog the works for spambots. It was clearly flagged as off limits in the robots.txt but Ask choose to ignore that. I don't want to send the message that it's okay to be a bad netizen and then later redeem yourself by virtue of making money (off of other people's content, no less). As far as I'm concerned, they were a bad bot then and they're a bad bot now and they can rot in hell. Also, even though my email address is clearly listed on perldesignpatterns.com, for some reason this... woman had to contact my publisher. What the hell? If someone is published once, the only valid way to reach them is through their publisher? They're crackmonkeys and everyone should ban them. Also, I probably firewalled a bunch of IPs of theirs to enforce the robots.txt ban when the robots.txt ban didn't cut it and I got tired of seeing hits from them scroll by and I don't want to wade through and figure out which IPs to unblock. Hey, maybe that's why she had to talk to my publisher. Also, the tone was whiney and unapologetic... if she had inquired about they had done in the past to reassure me of a resolution, I'd have more sympathy, but requesting the same access their competitors have just doesn't fly. Truth of the matter is bots already outnumber humans 10:1 and GOOG drives ten times more traffic than any other bot. If I start letting other bots in, it'll be 100:1 before I can blink.
Argument for unblocking them: She emailed me. That's about it.
Crud. Fuggit. I have better things to do. I already wasted enough of my time banishing them in the first place. It's their own damn fault. They run disrespectful bots, getting firewalled and shitlisted and having access to less content on the 'net is the natural result.
-scott
