All the Perl that's Practical to Extract and Report

Is your file tidy? Really?

ferreira on 2008-05-01T02:50:08

With respect to Perl::Tidy, it is quite obvious that a tidy file is one that has gone through perltidy and which does not change if reformatted again. 

$ perltidy tidy_file.pl
$ diff tidy_file.pl tidy_file.pl.tdy
# zero, no difference

But Perl::Tidy is a work in progress which is constantly improving. And improving means changing. Those changes may include the fixes of some behaviors considered non-ideal or even flawed. So tidy files under a Perl::Tidy version may not be tidy under another version. So this concept is a little fragile (or needs a more precise wording).

Even Test::PerlTidy (which intends to capture that tidy-ness concept in the form of an automated test) was caught by this detail, since a new release of Perl::Tidy got out. Edmund von der Burg and me are working on a reasonable approach to this issue. Wait for a new release of Test-PerlTidy soon.

Universal problem

ChrisDolan on 2008-05-02T02:27:30

I'm convinced that there is no "right" solution to this problem. Perl::Critic has the same issues, as does the CPANTS Kwalitee game. Coverity, FindBugs, PMD and CodePro all have the same issues in the Java world. Even the Acid3 browser test had this problem -- IIRC Safari (or was it Opera?) got 100% compliance, but then someone found a bug in the test and Safari lost their 100%.

Any test where you can achieve a perfect score on a subjective metric is destined to cause "not my fault" failures in the future.

For my Test::Virtual::Filesystem package, I introduced a feature where users could declare themselves compliant with a particular version of the test module. Then, I carefully increment the version of each test in the package when I make a new release, and if the test version number is greater than the claimed compliant version number then I make it a TODO test.

That approach works great for validating POSIX-like filesystems where correctness is quite objective. That approach would probably work for Test::Pod and would partially work for Perl::Critic (make new policies TODO failures == easy; deciding which changes to existing policies should be TODO vs. real failures == hard)

isa(Perl::Tidy, Silly)

educated_foo on 2008-06-01T03:52:59

Evaluating the human readability of code (i.e. its aesthetics) by how close it is to some machine-generated format is just wrong.

Re:isa(Perl::Tidy, Silly)

ferreira on 2008-06-01T13:02:02

I also believe that claiming a tidy file (as per Perl::Tidy) is more human readable is a too presumptuous statement. Sometimes perltidy bugs me too because I cannot make it fit exactly with my mental model of desirable code layout. But one thing it achieves: an automatic/objective way to determine a layout, cleaning gratuitous difference on code styles and fixing details like: avoiding tabs, enforcing cuddled elses everywhere, etc. And this is a good thing for large codebases, meant to be worked out by many developers (some of which we'll probably never have the chance of meeting personally).

This is as silly as the dumbness of other automatic mechanisms for code evaluation: like perlcritics, kwalitee, and even some tests which may verify formal behavior that has few or nothing to do with the expectances of the users about its API (for example, that happens a lot with CPAN distributions whose major interface components are scripts with little or no test coverage, which instead concentrates on the modules around the scripts themselves).