In a proof of concept for writing a better crawler, I noticed that lots of javascript content and dynamically generated links are lost. Cast around a bit and found the Perl module Javascript an interface to libjs
Then, I stumbled on HTTPUnit, an all singing, all dancing Javascript aware (they say there are limitations, of course) Unit testing framework. It's for Java, but it looks so promising that I think I will give it a shot on some link hiding site soon (perhaps this weekend ?)
