Cpantesters block MSN bot from accessing cpantesters site. It is said on blogs.perl.org that "Microsoft in their incompetent wisdom decided to unleash 20-30 bots every few seconds". Maybe it is, but closing access to a search engine is another side of incompetence, and I'd say hysterics and psychosis.
Aren't we wise enough to launch Ironman contest so that world search engines index blogging on Perl and not allow one of them to access the other Perl site? Restrictions should be smart; "mustdie" and blocking the whole range /^65.55.(106|107|207)/ are not the arguments at all.
Don't they think that by blocking MSN bot you block the site from indexing and later displaying in one of the search engines, thus by yourself and manually reduce the area where Perl-related stuff can be found?
P. S. And I can't comment on blogs.perl.org, as the site does not allow me either to register or to login.
It looks like yours was one of the accounts which got "half-made" when blogs.perl.org collapsed on its old software.
Your account was created, but never enabled because you didn't reply to the registration email. This is almost certainly because the process died before sending the mail.
I've now enabled your account so you should be able to log in and use the site. Please let me know (dave@dave.org.uk) if that's not the case.
For anyone else reading this. We know that there are a number of accounts in this state. It's easy for us to fix this problem. At the bottom of every page on the site there is a link for leaving feedback. Contacting us through that link is likely to be more effective that complaining on another blog.
Re:blogs.perl.org
andy.sh on 2010-01-19T14:20:06
Thank you, Dave. I'll try it
:)
If this was an isolated incident then yes blocking MS would be hysterics. However, they have been a consistence nuisance for the past year, and despite adding measures to reduce their tactics, they have decided to ramp up their use of the bots to the point it took out the site
The CPAN Testers server is something that a large part of the Perl eco-system relies on. Knocking it out of action is not something I am happy for anyone to do.
My reactions may have been emotional, but it has been a problem that has growing for a long time, and I'd had enough of having to monitor the server to make sure it didn't fall over because bots can't understand how to crawl a site politely.
The number of visits in the last month totalled 10,104, of that only 33 came from Bing related sites. These figures are taken from Google Analytics.
Re:The Bigger Picture
andy.sh on 2010-01-19T14:24:44
I can't be against any site inside Perl eco-system, of course, but anyway "time wget http://cpantesters.org/" prints 2 seconds response time which is not too cool.
It is said on blogs.perl.org that “Microsoft in their incompetent wisdom decided to unleash 20-30 bots every few seconds”. Maybe it is, but closing access to a search engine is another side of incompetence, and I’d say hysterics and psychosis.
So you would prefer that the site be almost inaccessible sooner than you would ban MSNBot?
This is not GoogleBot we’re talking about here, or even the Yahoo crawler. Who really cares about MSN? Honestly? I mean, do you use MSN search? Do you know anyone who uses MSN search?
Re:Errrm
andy.sh on 2010-01-19T14:18:48
> So you would prefer that the site be almost inaccessible sooner than you would ban MSNBot?
I think that this is a signal to improve site's performance, not more that that.
> Who really cares about MSN? Honestly?
That does not matter. If they with to collect data, they should be able to do that, I think. And are you sure that msnbot is not used for bing.com, which is default search engine in Windows 7's MSIE.
Re:Errrm
Aristotle on 2010-01-19T14:28:43
I think that this is a signal to improve site’s performance, not more that that.
You are aware that Barbie is doing the work in his spare time, right?
Re:Errrm
andy.sh on 2010-01-19T14:34:42
Oh my. I was not saying anything bad about Barbie, cpantesters, Perl, its community and MSN. My message was just "I think, it is wrong way to block search engine to cope with frequent requests".
Re:Errrm
Aristotle on 2010-01-19T14:52:40
That’s all very touching, but MSNBot is bombarding the site right now. What do actually propose?
Barbie should just allow MSNBot to continue to block the site for everyone.
Barbie should drop everything he was doing and work day and night until the site is fast enough for MSNBot’s abusive behaviour. (In the meantime, MSNBot is still blocking the site.
???
Doesn’t it seem more expedient to just ban MSNBot for the time being? I mean, no one said that the ban is supposed to be for all time. Eventually the site will be fast enough, and MSNBot will also hopefully learn respect at some point. In fact, Barbie’s loud complaints have caused other people to come out to say they have also had problems, and he has been contacted by someone from the Bing team. So the ban may get lifted sooner than expected.
Re:Errrm
mw487 on 2010-01-19T17:31:32
"If they with to collect data, they should be able to do that, I think."
I think so too, but they should not be evil when being nice can do the job.
And this banning is part of what is getting their attention. Barbie can not make them be nice, only they can decide to be nice. In the meantime, cpan should not bend over for them being stupid and mean and uncaring.
They know they have a problem, and they choose to go on anyway for their own profit.
Let them in when they can act properly.
I understand they ignore robots.txt. _Bad_
