All the Perl that's Practical to Extract and Report

Benchmarking perls

ethan on 2004-12-14T08:11:55

Partly due to the fact that I didn't have anything interesting to do, I wrote a little set of modules to benchmark perls against each other. There already is perlbench on the CPAN but I found the way tests had to be written inconvenient.

According to the results I get, realworld programs seem to get faster on recent perls. Some other things on the other hand are much slower, most notably regexes (almost by a factor 2 when comparing 5.5.4 with a threaded 5.8.6). The results:

Benchmarks     | perl5.5.4 | perl5.6.2 | perl | perl5.8.6 | perl5.8.6th | Weight
---------------+-----------+-----------+------+-----------+-------------+-------
bench/loops    |   1000    |    906    |  814 |    731    |     782     |     70
bench/regex    |   1000    |   1258    | 1978 |   1457    |    1977     |    116
bench/recurse  |   1000    |    961    |  983 |    956    |     991     |     73
bench/wave     |   1000    |    860    |  994 |    894    |     925     |    236
bench/autoload |   1000    |    968    | 1095 |   1078    |    1159     |    139
bench/substr   |   1000    |    947    |  984 |    986    |     957     |    164
bench/mail     |   1000    |    698    |  903 |    748    |     959     |    198
---------------+-----------+-----------+------+-----------+-------------+-------
Overall        |   1000    |    910    | 1085 |    960    |    1082     |   1000


The fourth column is the ordinary Debian testing-perl (perl5.8.4 with threads). The benchmarks can be found here. Some tests I consider almost irrelevant, namely autoload which is an OO version of the Fibonacci-number generator where each recursive instance is autoloaded. Also, loops is not interesting because empty loops are unlikely to show up that often. Real world programs are wave which decreases the volume of a 18meg WAV file, mail which walks through a 28meg mailbox and substr which calculates the length of the longest common substring of perldoc -tT perlfunc. It's 647 by the way, would you have guessed?

The rightmost column is the relative amount of time this test took on the leftmost perl. This number is used for calculating the weighted mean in the bottom row. In case the Weight column doesn't add up to 1000, that's due to some unclever rounding of my code.

Finally, the module which does the timing and that is included from each benchmark script is this: 
package Perl::Benchmark::Lib;

use strict;
use Time::HiRes;

use base qw/Exporter/;
use vars qw/$VERSION @EXPORT/;
$VERSION = '0.01';

$SIG{__WARN__} = sub {};
$SIG{__DIE__} = sub {};

@EXPORT = qw/bench_adjust/;

my $corrective = 0;

bench_adjust();

sub bench_adjust {
    my ($code, $times) = @_;
    if (defined $code) {
	$times ||= 1;
	$corrective += eval sprintf <
sub report_timing {
    print Time::HiRes::tv_interval(\@Perl::Benchmark::Lib::PB_timer) - $corrective;
    print "\n";
}

END {
    report_timing();
}

1;


It dumps the number of seconds of the runtime to stdout where it is picked up by another module that does all the bookkeepting and eventually spits out the table you see further above with the help of Text::Table. A very nice module, by the way.

And blead ?

rafael on 2004-12-14T10:03:55

Obviously a column is missing :)

Re:And blead ?

ethan on 2004-12-14T12:26:50

I was just lacking the patience to start up buildaperl and from there build a fresh bleadperl. But now that I have a fairly useful benchmark environment, I am quite sure I'll use it the next time I patch around in bleadperl.

I also would like to release it as Perl::Benchmark to the CPAN. Prior to that, it needs some polishing and maybe some more features. Also, I would like to create a benchmark suite that only relies on core-modules but is still real-world-ish enough to be useful. Maybe I find a way that it can run and time Perl test-scripts without modifying them. At least for those written with the Test-Harness framework, this should be possible.

Regexes

bart on 2004-12-14T10:47:54

Some other things on the other hand are much slower, most notably regexes (almost by a factor 2 when comparing 5.5.4 with a threaded 5.8.6).
If I may just take one uninformed guess on the cause: Unicode. Internal Unicode support was added starting with 5.6.0.

Re:Regexes

ethan on 2004-12-14T12:19:57

Internal Unicode support was added starting with 5.6.0.

Ah, that's plausible. I was also thinking of possible causes for this decrease in performance but didn't think of the obvious case.

And the additional slow-down caused by threads is then also fairly obvious: Perl's regex engine, which is very non-reentrant by nature, has to work more reentrantly now.

Other than that, I agree with mir's comment that with all the new features and bugfixes, the results are very favourable for perl and its porters. I always assumed perl had gotten slower (due to bloat) but for programs running in the wild (and not in a benchmark laboratory) the opposite is true.

Very impressive

mir on 2004-12-14T10:54:10

I think these numbers are very impressive. If you consider the increased features (Unicode!) and the bug fixing between 5.5.4 and 5.8.6, then keeping more or less the same speed is quite a feat. Kudos to the porters!