All the Perl that's Practical to Extract and Report

Failure Counts Update & Duplicate Reports

barbie on 2007-03-13T13:32:13

Following suggestions from others, I have now extended the Failure Counts pages of the CPAN Testers Statistics site. Firstly, more fields are exposed to highlight the date the PAUSE post appeared on the list, the total passes, total reports and the percentage of failures against total reports. Secondly, is that you can list according to failure counts or failure percentages by clicking the relevant column heading.

Some things to bear in mind though:

  1. The counts are based on the current monthly DB snapshot. I may change this at some point, but not right now.
  2. There are missing dates because the PAUSE mails have not been recorded on the NNTP version of the list. Not sure why this is, as they will have hit the mailing list, but I will investigate this when I have more time.
  3. The reports posted are not ignored where duplicates are concerned. In the case of an incident last night, this may obscure the true values. This requires a little thought, but I have an idea for that, which I'll come to.
  4. All CPAN Testers reports are based on the PAUSE Upload of the original author. I got an email the other day telling that the occurance of Scalar-List-Util-1.15 in the list was wrong, as 1.19 has several passes and 1.15 wasn't even listed for Scalar-List-Utils on the CPAN Testers site. Score yourself a point if it takes you less than 5 seconds to spot the faux pas. I don't have access to rewrite USENET, so unfortunately those will have to remain. Unless the original authors decide to make an upload to hide it from view ;)

The problem with duplicate reports though is problematic. Agreeing that a report for a distribution that matches the tester, platform and perl be considered a dupicate might be posible, but there may be good reasons why matching reports may occur. The same tester might have multiple setups for the same environment, such as one setup with no 3rd party libraries, while others might have specific library setups. It would be nice to allow the tester themselves to submit a report that indicates a specific submission entered previous was a mistake and should be ignored. This is most useful when a FAIL report is sent, only to discover that the test environment was incorrect, and subsequent fix and retest produces a PASS report. But how can a tester easily indicate a particular report should be ignored? There are couple ways I thought of:

  1. Send a report with the subject line 'IGNORE: 123456', where the number indicates the NNTP ID of the report. This requires effort on the part of the tester to find the report and for the software to verify that the sender of both matches. However this is open to abuse as the sender can be faked. Plus as with the PAUSE upload posts I've noted above, these may not make it through to the NNTP server for some reason.
  2. Send a email to an administrative address indicating the report(s) that are duplicates. The administrator can then verify the duplicate reports and provide an ignore report, similar to the first example, from a read-only feed, e.g. a web page, RSS or API, which can then be read by software tools to ensure that those listed are considered duplicates.

At the moment I believe only CPAN Testers and CPAN Testers Statistics would make use of this, but in the longer term, it may make the reporting mechanism a little more reliable, and others may have more confidence in the reporting information that is provided.

Personally I think the second is the better option, but it does require a lot of changes and implimentation. Rather than rushing ahead and trying to solve the problem, I thought it better to make the suggestion and see what others think. Maybe there is an even better solution I've not thought of. So ... thoughts welcome :)

Use the message-id, Barbie

drhyde on 2007-03-14T11:22:10

Both email and usenet messages have unique message-IDs. I presume that when a report hits the cpan-testers list and gets copied to the nntp version of the list, the message-id is preserved. So the simple solution is to accept subject lines like "CANCEL <msgid123456789@example.com>".

Or let people cancel their results interactively through a web shite.

Re:Use the message-id, Barbie

barbie on 2007-03-14T13:37:17

Whether it uses the message-ID or the NNTP id is secondary, it's whether the cancellation is from the original tester that's the important bit.

Cancelling through a website would be better, but it still needs a method to verify that the cancellation request came from the original tester.

I would expect cancellations to only happen occasionally, so having an administrator just verify the request wouldn't be too bad. If it got to be a regular occurance then it would likely indicate a bug that needs fixing before testing continues.