Dear Log,
The other day I said on the Web-Robots list that I was having trouble with a particular feature in LWP::RobotUA and that I suspected that WWW::RobotRules's algorithm for matching the user-agent's name against robots.txt rules was perhaps maybe just a bit buggy in one particular circumstance maybe kinda just a bit. The response from several people on the list was basically that it was quite unthinkable that there could be a bug in LWP!
I wonder how you get people to think things like that about your software. Hypnotism? Pheromones? Blackmail?
So I explained in excruciating detail (which I had earlier assumed would be unnecessary) exactly the conditions under which the code misbehaved. Everyone then said "hm, well then, hm, maybe that is a bug."
Of course, fixing it was just a matter of my sending Gisle a tiny patch, thus making the Web a better and more standards-compliant place.
And this afternoon, in lieu of doing anything really productive like working on my chapter, I make sure that LWP::RobotUA really behaved as advertised in other respects too. I found some more bugs (like: it thinking that all URLs have to be screened thru a robots.txt, not just "http:" / "https:" ones). Patches composed and submitted. That didn't waste enough time, so I took the opportunity to spiff up Gisle's documentation.
Meanwhile, Linda Mui at O'Reilly (mmmmuy linda) sent me some draft manuscripts: HTTP: The Definitive Guide and Perl & XML. I've only looked at the first one, for just a while. It seems absolutely brilliant. I was pleased to see that it covers all sorts of issues to do with writing large-scale robots -- which I take to mean that I don't have to! I'd have felt quite guilty if my forthcoming book were to be the only book out there mentioning spiders, while failing to go into the extreme detail that the subject of problems facing large-scale spiders deserve.
Whee, and as I write this, I just got email from Powells.com saying that my order has shipped! The order consists mostly of books that others have recommended to me:
BTW, thanks to everyone who replied to my earlier post. Just seeing people react to what I was writing managed to jump-start my brain.
