As I've recently taken over a distribution relating to the CPAN Testers, I've been looking to increase it's test coverage. I have several more samples and tests to add to the suite, and wanted to see what the results were. The following are all using the lastest version, 0.64, from either CPAN or a PPM repository.
The first extract is on Windows, from a PPM installation, as I don't have a compiler on the machine. The results are annoying and misleading. Annoying as it's choosen to include everything bar the kitchen sink. According to the docs, you can use -ignore and -inc to get rid of these, only -ignore doesn't work and I get a 'non-existant option' error for -inc. It's misleading as the two items at the bottom of the table are the bits I'm interested in, however, the results are useless as there is nothing to indicate what the real coverage is for either. The second to last line isn't the coverage for that module.
C:\wip\distros\CPAN-Testers\CPAN-WWW-Testers-Generator-0.23>cover Reading database from C:/wip/distros/CPAN-Testers/CPAN-WWW-Testers-Generator-0.2 3/cover_db Devel::Cover: ignoring extra subroutine Devel::Cover: ignoring extra subroutine Devel::Cover: ignoring extra statement Devel::Cover: ignoring extra statement Devel::Cover: ignoring extra branch Devel::Cover: ignoring extra branch Devel::Cover: merging data for lib/CPAN/WWW/Testers/Generator/Article.pm into bl ib/lib/CPAN/WWW/Testers/Generator/Article.pm ---------------------------- ------ ------ ------ ------ ------ ------ ------ File stmt bran cond sub pod time total ---------------------------- ------ ------ ------ ------ ------ ------ ------ .../lib/ActivePerl/Config.pm 0.0 0.0 0.0 0.0 0.0 n/a 0.0 ...l/lib/ActiveState/Path.pm 0.0 0.0 0.0 0.0 100.0 n/a 4.8 C:/Perl/lib/AutoLoader.pm 0.0 0.0 0.0 0.0 0.0 n/a 0.0 C:/Perl/lib/B.pm 14.6 16.7 16.7 12.5 n/a 93.6 15.0 C:/Perl/lib/B/Debug.pm 0.0 0.0 0.0 0.0 n/a n/a 0.0 C:/Perl/lib/B/Deparse.pm 0.1 0.0 0.3 0.2 0.7 1.8 0.2 C:/Perl/lib/Carp.pm 10.5 0.0 n/a 22.2 0.0 0.0 9.8 C:/Perl/lib/Carp/Heavy.pm 6.6 0.0 0.0 16.7 0.0 0.0 4.7 C:/Perl/lib/Config.pm 52.6 40.0 n/a 40.0 n/a 0.0 47.1 C:/Perl/lib/Config_heavy.pl 0.0 0.0 0.0 0.0 66.7 n/a 4.1 C:/Perl/lib/Cwd.pm 3.3 0.8 0.0 11.1 25.0 0.0 2.6 C:/Perl/lib/Data/Dumper.pm 0.0 0.0 0.0 0.0 25.0 n/a 1.1 C:/Perl/lib/Digest/base.pm 0.0 0.0 n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/DynaLoader.pm 70.0 50.0 66.7 50.0 33.3 1.0 61.4 C:/Perl/lib/Errno.pm 39.4 0.0 0.0 50.0 n/a 0.0 29.0 C:/Perl/lib/Exporter.pm 52.1 71.4 56.8 44.4 0.0 0.3 52.2 ...erl/lib/Exporter/Heavy.pm 14.8 16.3 31.3 33.3 0.0 0.2 16.7 C:/Perl/lib/Fcntl.pm 0.0 0.0 n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/File/Basename.pm 28.9 18.1 9.1 60.0 100.0 0.0 24.9 C:/Perl/lib/File/Find.pm 0.0 0.0 0.0 0.0 25.0 n/a 0.2 C:/Perl/lib/File/Glob.pm 0.0 0.0 n/a 0.0 20.0 n/a 1.1 C:/Perl/lib/File/Path.pm 70.2 48.8 29.4 100.0 0.0 0.0 53.5 C:/Perl/lib/FileHandle.pm 0.0 0.0 n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/IO.pm 0.0 0.0 n/a 0.0 n/a n/a 0.0 C:/Perl/lib/IO/File.pm 59.1 36.4 28.6 66.7 100.0 0.0 46.9 C:/Perl/lib/IO/Handle.pm 7.2 2.6 11.1 10.3 21.6 0.3 8.0 C:/Perl/lib/IO/Seekable.pm 0.0 0.0 n/a 0.0 100.0 n/a 16.7 C:/Perl/lib/IO/Socket.pm 14.1 1.0 1.9 25.8 41.7 0.0 11.5 ...erl/lib/IO/Socket/INET.pm 12.4 0.0 0.0 27.3 70.0 0.0 8.4 ...erl/lib/IO/Socket/UNIX.pm 30.0 0.0 0.0 42.9 75.0 0.0 20.0 C:/Perl/lib/MIME/Base64.pm 100.0 n/a n/a 100.0 n/a 0.0 100.0 ...l/lib/MIME/QuotedPrint.pm 100.0 n/a n/a 100.0 n/a 0.0 100.0 C:/Perl/lib/Net/Cmd.pm 4.5 0.6 0.0 15.4 66.7 0.0 6.4 C:/Perl/lib/Net/Config.pm 42.9 0.0 n/a 80.0 100.0 0.0 37.0 C:/Perl/lib/Net/NNTP.pm 6.9 0.0 0.0 8.4 90.2 0.0 8.1 C:/Perl/lib/Pod/Find.pm 0.0 0.0 0.0 0.0 100.0 n/a 1.3 ...l/lib/Pod/InputObjects.pm 0.0 0.0 0.0 0.0 n/a n/a 0.0 C:/Perl/lib/Pod/Parser.pm 0.0 0.0 0.0 0.0 0.0 n/a 0.0 C:/Perl/lib/SelectSaver.pm 0.0 0.0 0.0 0.0 0.0 n/a 0.0 C:/Perl/lib/Socket.pm 56.3 10.0 0.0 75.0 100.0 0.0 43.1 C:/Perl/lib/Storable.pm 0.0 n/a n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/Symbol.pm 10.8 0.0 0.0 16.7 0.0 0.0 6.3 C:/Perl/lib/Win32.pm 26.9 0.0 0.0 16.7 6.7 0.0 15.8 C:/Perl/lib/XSLoader.pm 66.7 50.0 22.2 50.0 50.0 0.2 56.5 C:/Perl/lib/base.pm 39.5 35.7 36.8 57.1 0.0 0.2 37.6 C:/Perl/lib/blib.pm 72.2 37.5 33.3 100.0 n/a 0.0 60.0 C:/Perl/lib/bytes.pm 0.0 0.0 n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/constant.pm 38.0 22.2 25.0 25.0 n/a 0.0 30.4 C:/Perl/lib/integer.pm 50.0 n/a n/a 50.0 0.0 0.0 40.0 C:/Perl/lib/lib.pm 57.9 26.9 33.3 83.3 0.0 0.0 48.5 C:/Perl/lib/overload.pm 0.0 0.0 0.0 0.0 18.2 n/a 1.3 C:/Perl/lib/re.pm 0.0 0.0 0.0 0.0 0.0 n/a 0.0 C:/Perl/lib/strict.pm 84.6 62.5 50.0 100.0 0.0 0.0 71.4 C:/Perl/lib/utf8.pm 0.0 0.0 n/a 0.0 0.0 n/a 0.0 C:/Perl/lib/vars.pm 44.4 45.5 0.0 100.0 n/a 0.2 40.4 C:/Perl/lib/warnings.pm 10.5 7.1 0.0 12.5 50.0 0.0 9.1 .../lib/warnings/register.pm 100.0 50.0 n/a 100.0 0.0 0.0 89.5 .../lib/CPAN/DistnameInfo.pm 68.3 57.7 11.1 41.7 81.8 0.0 54.6 ...ite/lib/Class/Accessor.pm 35.3 17.6 15.2 30.4 85.7 0.2 32.0 ...ib/Class/Accessor/Fast.pm 43.5 30.0 n/a 50.0 100.0 0.0 45.5 ...rl/site/lib/DBD/SQLite.pm 34.8 11.9 0.0 53.8 0.0 0.2 21.6 C:/Perl/site/lib/DBI.pm 19.3 8.3 7.6 27.4 n/a 0.2 14.4 ...site/lib/Devel/Symdump.pm 0.0 0.0 0.0 0.0 100.0 n/a 2.2 .../site/lib/Email/Simple.pm 91.4 50.0 40.0 94.4 100.0 0.0 85.0 ...ib/Email/Simple/Header.pm 35.2 23.7 16.7 38.5 100.0 0.3 33.5 ...ite/lib/File/Spec/Unix.pm 0.0 0.0 0.0 0.0 100.0 n/a 6.6 ...te/lib/File/Spec/Win32.pm 18.7 11.4 0.0 23.1 100.0 0.0 20.1 .../site/lib/Pod/Coverage.pm 0.0 0.0 0.0 0.0 100.0 n/a 2.8 .../Coverage/CountParents.pm 0.0 0.0 n/a 0.0 n/a n/a 0.0 ...l/site/lib/Scalar/Util.pm 0.0 0.0 0.0 0.0 50.0 n/a 2.2 .../site/lib/Test/Builder.pm 49.6 26.8 26.3 65.2 100.0 0.5 44.4 ...ib/Test/Builder/Module.pm 90.6 50.0 33.3 100.0 100.0 0.0 86.7 ...te/lib/Test/MockObject.pm 42.0 14.3 26.8 36.5 84.6 0.0 39.1 ...erl/site/lib/Test/More.pm 24.7 7.5 4.8 36.6 95.5 0.0 22.6 ...rl/site/lib/Time/Local.pm 38.4 0.0 2.8 60.0 100.0 0.0 25.0 .../WWW/Testers/Generator.pm 90.5 58.3 69.2 100.0 100.0 1.1 81.7 ...ters/Generator/Article.pm n/a n/a n/a n/a n/a n/a n/a Total 15.5 7.2 5.8 19.2 30.3 100.0 13.0 ---------------------------- ------ ------ ------ ------ ------ ------ ------
So okay that was Windows. Just to see what would happen, I copied the code across to a Linux box and reran the coverage tests. The following is what I get. All the lines about core or installed modules have disappeared, but I still get the misleading lines. In fact despite the same code being run, notice the numbers are different for the Generator.pm line.
---------------------------- ------ ------ ------ ------ ------ ------ ------ File stmt bran cond sub pod time total ---------------------------- ------ ------ ------ ------ ------ ------ ------ .../WWW/Testers/Generator.pm 89.4 57.8 75.0 100.0 100.0 100.0 81.1 ...ters/Generator/Article.pm n/a n/a n/a n/a n/a n/a n/a Total 89.4 57.8 75.0 100.0 100.0 100.0 81.1 ---------------------------- ------ ------ ------ ------ ------ ------ ------
After playing with the test files a bit, I discovered that removing one specific file, produces the following results:
---------------------------- ------ ------ ------ ------ ------ ------ ------ File stmt bran cond sub pod time total ---------------------------- ------ ------ ------ ------ ------ ------ ------ .../WWW/Testers/Generator.pm 28.9 4.7 8.3 58.8 100.0 7.7 24.4 ...ters/Generator/Article.pm 96.1 58.7 42.9 100.0 100.0 92.3 81.9 Total 60.6 27.3 32.5 78.1 100.0 100.0 52.1 ---------------------------- ------ ------ ------ ------ ------ ------ ------
Now that's more like it. However, the one file that I removed contains a large part of the tests for the first module. The only thing I can think that might be causing this is that I'm using Test::MockObject to avoid network usage. There is nothing special about the test file, it runs some samples through the system and creates a couple of files. I would consider submitting this as a bug, both for the spurious inclusion of installed modules and the dodgy coverage lines, but is this a bug, or something about the way I'm using Devel::Cover? Does it have a problem with Test::MockObject? Is there a way to suppress any recording of installed modules?
I've now edited the offending test script, editing out any reference to Test::MockObject and I still get the misleading lines, so it isn't that. I'll carry on writing tests, but it's annoying that I can no longer use Devel::Cover to see what I'm missing :(
I've tried searching Google and Perlmonks for posts about similar issues and haven't found anything. Have I really found an issue (or rather issues) that no-one else has? If you've come across these issues, please let me know, preferably with links to solutions, otherwise if they are new, I'll try and figure how better to submit them to RT.
...but I find it's something people often don't know, and it made me crazy until pjcj explained it to me.
Basically @INC is cached at install time. If you update perl, you need to re-install Devel::Cover. If that's not working or you can't do it, inspect Devel::Cover::Inc.
Re:maybe you know this...
barbie on 2008-07-16T21:14:23
I didn't know that
:) Unfortunately I can't reinstall, as I don't have a C compiler on the Win32 machine (and without going into details it's not an option at the moment). However, I've had a look at Inc.pm as you suggested and altered the paths by hand and .... yeah! It works. Lovely. This would be a good entry for the docs. Will figure out a patch for that :) Now I just need to figure out why that 1 test script causes the other problem.
Thanks.
