What do you do when people keep doing the wrong thing? I see three general types of responses:
Those are in the order in which they should be applied. The problem is that item 3 is considered heresy by some people. It's such an awful heresy that sometimes when I bring it up publicly, I get soundly chastised for this. Sometimes I deserve it and it forces me to rethink my position. Sometimes I ignore the criticism and the people who use my solutions are happier for it. Data::XML::Variant is a case in point. I hope you never have to use that steaming pile of ones and zeroes, but if you are sending, say, IDIF feeds to Yahoo! in their pseudo-XML format, you're not going to get them to fix things and you know it, so option #3 is on the table.
Now I have a problem with our test suite at work. If we run the "fast" version of our tests, they take about 7 to 8 minutes. The regular version of our tests takes 35 to 45 minutes. Lately it seems that developers are running the fast version of our tests before merging branches to trunk and not running the slow tests. As you might expect, our trunk has broken a few times.
We use Hudson for an integration server, so we find out about this, so it's not the end of the world, but I'm trying to figure out how to solve this.
The "fast tests" have saved us so much time and agony that they're "making the wrong thing (not running all the tests) less wrong" and we get more work done, but how to get the developers to run the full test suite before merging to trunk? Once again I need to go back to step 1 (do the right thing) and then to step 2 (make it easy to do the right thing) before moving to step 3.
Welcome to the real world: sometimes it's not nirvana.
It sounds like your version control system could help you here. It should be easy to use Devel::Cover to generate a source file -> relevant test mapping. From there, you can see which files are affected by a merge. Then, you can compute a minimum set of relevant "slow" tests to run. This should prevent the merger from having to run all the tests, and save him some time.
Obviously this could go wrong in a number of ways, but it might be helpful anyway.
Re:Making it "less wrong"?
Ovid on 2008-12-15T16:40:31
Devel::Cover is a wonderful tool, but we have a lot of trouble using it. Our test suite is slow enough, but with Devel::Cover, it can only be run once, and that's an overnight run (7 to 10 hours).
What this means is that we can only run it on trunk, but we need it run on our branches. Since they change, add and (sometimes) delete tests, we've no real guarantee of which tests from a branch should be run, thus putting us back at square run.
And for extra credit, create a multi-user development environment where people can set up their own PERL5LIB (since they need to be able to play with new CPAN modules and not break other users) and then run this:
perldoc -m Devel::Cover::Inc
Yeah, it's heart-breaking. That information is hard-coded in there from information gathered at the time Devel::Cover was installed. I love Devel::Cover, but it needs some TLC.
Re:Making it "less wrong"?
Ovid on 2008-12-15T16:42:43
thus putting us back at square run
WTF? "Square one".
Re:Making it "less wrong"?
jplindstrom on 2008-12-15T17:54:38
Devel::CoverX::Covered does the mapping (which turned out to be far from "easy").
But it doesn't quite solve the problem.
Which isn't really a technical problem anyway. People just needs to run the tests.
Re:Making it "less wrong"?
autarch on 2008-12-15T20:28:52
Except it is a technical problem because the tests take 40 minutes. That's a lot of developer time wasted.
Re:Making it "less wrong"?
Ovid on 2008-12-15T21:54:11
Yes, this is a huge problem. It's also a hell of a lot faster than it was. Huge amounts of work went into speeding up that test suite and Test::Aggregate is one of the results.
This doesn't make the wrong things wrong-er but it does make the right thing easier.
