After successfully mapping 54 addresses from September up to mid-October, I sent out the six-month request to all outstanding addresses used so far this year. After only a short time I'd already received several emails from testers updating their details for me, mapping a further 14 addresses. It is notable that several new and current testers are now putting their names, or a least some form of reference in their addresses, that makes it very easy for my scripts to reference them, so thanks very much for that.
One thing that does still irritate me though is testers who use an illegal, malformed or unreacheable address. Aside from the stats, if as an author I receive a rather unusual test result and want to follow it up, who do I contact? 'root@dmz1' or 'root@bender.domain' is of no used whatsoever. If you're going to get involved with CPAN Testers, ** PLEASE ** use a legitimate and accessible email address so that authors can contact you if they need to.
Following on from that, I can understand that some may want to protect their email address from spam. However, use some mechanism that a human can detect easily. Using 'spamproof@nospammail.net' is not obvious. Do I remove 'nospam' to get '@mail.net', it would seem not, as the original address is legitmate and resolves to a legal domain, except it's used to blacklist email addresses. Genuis! So authors who respond to you get blacklisted. Thanks!
If you are a current tester and use email addresses like this, please change them so you can be contact by authors and not have communication end in a black hole. It has already been proposed that testers should register, somewhat more officially, that they wish to become testers, which the more I think about it, is probably the right way to go. We currently have a large number of testers regularly posting reports, even if some are only sending 2 or 3 a month. Those that send with badly configured test environments, we should be catching and encouraging to improve their setup if we can. Registering your interest might be a better way to enable that.
Thanks to the efforts of our top 4 testers, my nightly scripts now take nearly 10 hours to run at the end of the month! The basic update takes roughly 2-3 hours and then the backend analysis and verification takes anything up to 8 hours to run at the moment. I'm looking to improve these as I'd like to be able to automate the publishing of the stats on a daily basis. The database is already published daily, so it shouldn't be too difficult. However, the verification process enables me to check that any uploads and reports haven't been missed or badly formatted, so that I can send the authors or testers emails to alert them to any problems. By improving the automation, I'm hoping this can be done rather more quickly to raise the alerts earlier.
In mid-October Andreas was looking to take over Chris' monthly top tester spot, having submitted twice as many as Chris at the time. Since I happened to mention it to Chris when I saw him the other week, Chris has taken back his regular lead. Andreas has now taken up 3rd position on the leader board, and making a good attempt to take 2nd place. Slaven Rezic is also making notable headway with his testing setup, to the point where he is now the 14th tester to have submitted over 10,000 reports, jumping from 24th to 10th place. Well done Slaven. The other notable mover is Nigel Horne who has moved from 11th to 9th. Several new testers have joined the crew, with both Matthew Musgrove and Phil Monsen making a a very healthy debut, to the point we had 128 testers during October. Overall we have 600 known testers and 546 unmatched email addresses, making up 1146 listed testers at the moment, with only 10 unmatched email addresses for last month.
If you want to get involved in CPAN Testing, please checkout the CPAN Testers Wiki for details of how to set things up. You can also join the CPAN Testers Discussion mailing list if you need any further advice or if you have any suggestions for improving the tools or resources.
I should probably add email validation to CPAN::Reporter. This wouldn't catch "nospam@dontspamme.com" but would at least ensure a valid RFC format. Do people think that's useful or is this pointless?
If pointless, what would we need to have it be relevant? Send a test message and see if the MX accepts it? That's going to be a pain to do reliably because of firewalls. Just check that there is an MX for the given domain? Is that enough?
And which CPAN modules are back-compatible enough to add as a pre-requisite? (5.005 at least, with wide platform support.)
Other ideas?
-- dagolden
Re:Email validation
rjbs on 2007-11-02T22:51:08
Let people register as testers by confirming an email address that is used as their "from." Then it can be made clear which reports come from registered testers. More data could be collected, if they were willing: contact prefs (STFU or "happy to help"), location, platforms tested, etc.Re:Email validation
drhyde on 2007-11-03T18:43:02
Unfortunately, hlagh@hlagh and hlagh@hlagh.hlagh are perfectly legal according to the relevant RFCs. Some form of registration system would work to fix this, and would also presumably be part of the shiny HTTP future and so reduce the mail load on perl.org. But it would also form a barrier to entry for those who just submit test results for the small number of modules they need for their own useRe:Email validation
barbie on 2007-11-04T13:06:16
As Dave mentions, whether an address is RFC compliant doesn't guarantee a real address. Whether the domain resolves to an MX record again doesn't mean the address exists. The only way is to send a mail to the user requesting a response to verify themselves, such as happens with several mailing lists. With so many people getting involved with testing now and the value of ensuring authors can contact testers should they need clarification of any reports, then registration is a logic next step.
With the HTTP interface in place then allowing testers to register and authors to set preferences, would likely be seen as a good move forward.
CPAN Testers has moved on a lot since it was first instigated and it's probably about time there was some formal acknowledgement of the testers involved. Hopefully those testers who only submit low volumes of reports wouldn't be put off by the initial setup process though. Unless we allow an anonymous account.
Since CPAN::Reporter I have toyed with the idea of becoming a tester… however, all I have to test on is my dev machine, a bog-standard Slackware 12.0 machine, ie. an unthreaded 5.8.8 on 32-bit x86 Linux, with my particular mix of required modules installed among the system perl. I wouldn’t set up any sort of controlled environment. I wouldn’t be going out of my way to generate test reports either – just send the occasional one when I install something.
And I have to wonder if that’s at all interesting to anyone. Surely the sort of environment in which I test is pretty common? It seems to me that this is exactly the sort of setup that many Perl developers already use themselves (so they probably don’t need any test reports to begin with) and which is well represented among existing testers (so I’d just be submitting me-too reports).
I do not have an unusual platform to test on, or at least some sort of controlled environment (eg. a fresh perl for each module under test with no non-core modules pre-installed). But it seems to me that the core value of testing is to provide feedback about behaviour in those controlled or exotic environments that not everyone and their dog and brother already has access to.
Am I missing something here, or is that right? Would I contribute something of value if I were to get involved at the initially described low level of participation?
Re:What testing is still needed?
barbie on 2007-11-04T12:55:07
Occasionally, for whatever reason some distributions do miss out. Usually because they require libraries that the testers don't have installed, or they filter out some distributions due to problems with test them (e.g. I couldn't test Apache modules on Win32). So even if you do have a common setup, it is still possible for you to find a failure that hasn't been found by other testers. Whether you only submit 1 report a month or hundreds, it all helps
:) Also because you don't have a minimal "fresh" installation environment like the prolific testers, you are more likely to experience issues that regular users would experience with installations, which are of value to authors too.
