From Tim Bunce's Perl 5 Myths talk: Perl 5.10.0 has 78,883 core tests.
Ruby has about 1,400.
1,400.
So Perl 5 core test suite has about 50 times as many tests as Ruby. So who has the testing culture again? It's not entirely a fair comparison because it's not asserting how many features each language has, but I doubt that Ruby is so feature poor to justify Perl having more than 50 tests for every 1 of Ruby's.
Hello, core Ruby developers! You've got an awesome language there. Could you start testing, please? Your community seems to do OK. Why not you?
What happens if you add CPAN tests to that count? And whatever is the Ruby equivalent.
Re:CPAN?
Ovid on 2009-04-17T15:25:34
Tough to say. According to Tim's slides, if we include bundled libraries, we drop significantly to a 7 to 1 advantage for tests. However, there's no clear way I can see to design a reasonable metric for non-core tests. LOC/test is silly, particularly since core Ruby seems to generate terser code than core Perl for similar functionality. Tests per function point would be difficult to analyze without static analysis of function points and agreement on said analysis.
To be fair, even the core tests might be problematic if they bundle multiple assertions into a single "test". However, they'd still have to have an average of over 50 assertions per "test" and that would suggest that their tests weren't granular enough.
Also, assessing the quality of the tests is impossible. Even 100% coverage is a pretty silly metric with only unit tests and full path coverage testing is impossible.
More importantly, the CPAN tests would be almost useless even if we did have agreement. Class::Trait has tests to assert that we didn't fall afoul of the SUPER:: bug, so if core Perl had no test for that, this merely means that we test that certain unwanted behaviors don't impact our code, not that Perl doesn't have these behaviors.
Core tests are far, far, far more important than non-core tests. You need to know that the foundation you're building on is rock-solid and without core tests, it's tough to be certain of that, particularly since a well-designed, stable test suite can almost be viewed as a contract, something Ruby currently lacks.
I know there are Ruby developers who are very frustrated with this (Daniel Berger is one, if I recall correctly) and have complained in their blogs that the core Ruby developers aren't really taking this as seriously as they should.
50 to 1 test coverage kind of makes that point without further argument, but it would be nice to know if such an important technology is coming up to speed in this area.
Re:CPAN?
dagolden on 2009-04-17T15:38:24
Was thinking also about things like CPAN Testers. 3,592,446 public test reports (as of last night) vs ???.
At least as evidence of a "testing culture".
:-) -- dagolden
Re:CPAN?
Ovid on 2009-04-17T15:45:46
Heh
:) I'm unaware of any other language community having anything remotely comparable.
Re:CPAN?
Alias on 2009-04-17T17:08:04
... yet.
Many of the repositories are within a few years of us, so it's important to not get too proud about what we have, and keep pushing ahead.
Re:CPAN?
chromatic on 2009-04-17T19:25:51
First, those language communities have to get serious about including systematic automated testing throughout their repositories. I hope they do, but I haven't seen it.
Re:CPAN?
Aristotle on 2009-04-17T20:13:49
That doesn’t really affect Adam’s point. Most of the infrastructure we have was built by individual efforts, so other communities with enough sufficiently determined members could clearly duplicate it in a reasonable time frame. (My gut estimate is about 3 years all told.)
Even if other communities show no signs of such an effort and seem to be parsecs behind, that doesn’t mean we shouldn’t still keep pushing forward as if the would-be competition were breathing down our necks.
Even if that image falsely assumes that this is a competition in the first place. The point is, resting on laurels is a good way to dry them up; we can’t go wrong by continuing to push the envelope as hard as possible.
Re:CPAN?
chromatic on 2009-04-17T21:05:42
Even if that image falsely assumes that this is a competition in the first place.
I don't believe it is. I want to see a comprehensive and freely distributable and reusable test suite for every widely used language. That's the only way we can port them to Parrot, for example. (There's also the code quality issue. Can I assume everyone understands that by now?)
Re:CPAN?
jpedrosa on 2009-04-17T21:55:26
Thanks for the Perl 6 numbers.
Speaking of more languages to Parrot, how is JavaScript coming along? Or ECMAScript?
Because Google's v8 engine is a pretty good indication that JavaScript is a potential competitor for the server side needs as well. I have been creating some benchmarks and it fairs quite well against Ruby 1.9, Python and PHP, and while I don't like reinventing the wheel or the code bloat of JavaScript when compared to Ruby, it's certainly a platform to more than watch.
See k7: http://github.com/sebastien/k7/tree/masterRe:CPAN?
Aristotle on 2009-04-17T22:34:45
Oh, I’m not talking about the size of the language core test suite. I’m talking about the things Adam appeared to be referring to, by virtue of the comment which he replied to – namely the CPAN Testers and all the attendant infrastructure.
Re:How about Perl 6's numbers?
chromatic on 2009-04-17T20:59:50
Rakudo knows about almost 16,000 spec tests and passes just over 10,000. The total number should cross 25,000 eventually. (When Rakudo passes 16,000 spec tests, it'll be very usable.)
You should really check out Rubyspec: http://rubyspec.org/
These are the numbers from a CI run I did just a second ago: 2541 files, 10296 examples, 33053 expectations, 0 failures, 2 errors
Comparing test numbers is a load of crap, and you know that. It's the sort of stupid advocacy that people complain about when applied to Perl and shouldn't be used to compare Perl favorably to anything else. It's really embarrassing when big Perl names spread this sort of shit around. It hurts the credibility of people who try to advocate Perl in a fair manner. Ruby doesn't have to lose for Perl to be useful.
If you want to make a valid point, talk about test coverage. You know this, though. You went for a cheap shot though, and that helps no one.
Let's not forget, that despite all its tests, Perl 5.10 didn't have enough to ensure smart matching worked.
Re:Cheap shots
Ovid on 2009-04-20T07:21:33
You know, you could easily have phrased that without saying "shit" and "load of crap". It's really embarrassing when big Perl names act like this.
Re:Cheap shots
petdance on 2009-04-23T19:31:37
And "stupid."Re:Cheap shots
chromatic on 2009-04-20T19:04:43
Ruby doesn't have to lose for Perl to be useful.
Who says Ruby has to lose? Who says anyone wants Ruby to lose? (Who even knows what that means?) I'd like to see Ruby much better tested. It's a nice language, but I've found its lack of a systemic testing culture very frustrating.
It's entirely possible that 'test' means different things for Ruby and Perl. I recently looked into what it would take to port a particular module to Ruby (because, as far as I can tell, there's no solution for parsing RTF for Ruby), and if I'd done so I would probably have had less than a tenth as many tests. The test suite for the module tested a huge number of things for no reason that I could see, such as the contents of various internal buffers during the parsing process rather than testing the output (and I'd think that that would make refactoring almost impossible....).
I'm not saying that's the case here, or that Ruby has as much test coverage as Perl, but it's entirely possible. Does it matter whether Ruby has 1 test, 1,000 tests, or 100,000 tests if they all result in the same code coverage?
