A common response to "why aren't you testing" is "we have legacy code that's hard to test"? Having been in that situation, I can feel your pain, but you can still test if you think about it. The main thing is to understand how your customer views your code. Consider a typical Web site. They don't see a bunch of global variables, 500 line functions and hard-code HTML 3.2 embedded in said functions. They see a form that they fill in and submit. It's easy. So that's what you test.
my $mech = Test::WWW::Mechanize->new;
$mech->get_ok($login_url, "We should be able to get $login_url");
$mech->submit_form_ok({
form_number => 1,
fields => {
username => $username,
password => $password,
},
}, "... and we should be able to login to our Web site");
$mech->title_is("Welcome to example.com, $fullname");
Congrats. You've just written your first test. You've bypassed all of your worry of globals, 500 line subroutines, embedded HTML, and so on. And you know what? There's a good chance that the above tests have verified a heck of a lot of code.
This doesn't mean that your particular situation is this easy, but if you think hard enough, you can solve this problem. You just need to be creative.
You start writing tests like that and after a while, you can have a pretty comprehensive test suite and start feeling comfortable about refactoring. Fear-driven development becomes a thing of the past.
I agree that you should try your hardest to at least have *some* sort of regression testing in place.
But the problem with the above approach is that it's fragile. I know, because one of the codebases I work with has a *lot* of tests like that. And they're very noisy. So noisy that they tend to get ignored. And most cases where a test complains, it's some environmental or transient issue, or a false positive (i.e. someone has made a change that breaks the tests but hasn't introduced a bug).
This is compounded by the fact that running these tests can be very slow, so it's usually only done nightly. You might run a few of the tests around the code you've changed, but the ones you've broken aren't obvious ones to check. Tracking down the issue can often take a long time.
Not all of that is necessarily a symptom of using the approach you suggest (there are a lot of other issues with the test framework we're using), but I think you perhaps underestimate how hard it can be to get it right. There are some places in the codebase where the tests are worse than useless. On top of that, less senior developers are completely put off the idea of testing, since it's such a hassle for them.
I tend to think you're better off in the long run refactoring and writing unit tests as you go.
Re:Hmm...
Ovid on 2008-07-21T10:34:04
Not all of that is necessarily a symptom of using the approach you suggest (there are a lot of other issues with the test framework we're using), but I think you perhaps underestimate how hard it can be to get it right.
I can assure you that I've been down this road many, many times. For most companies I start with, I almost immediately find out that if they have a test suite, it's usually broken or limited in very fundamental ways. If they don't have a test suite, they always have a wide variety of excuses, few of which really hold water.
You are correct that the approach I outline can be problematic at times, but the advantage that such integration tests give you that unit tests do not is twofold:
- You can write the tests, whereas unit tests can be much more fragile
- You get decent coverage of user expectations, something that unit tests can't do.
For one company I worked with, I couldn't refactor code safely because there was no test suite, but when I found myself facing 500 line functions with global variables galore, falling back on Test::WWW::Mechanize allowed me have a bit of confidence that at least I wasn't doing anything catastrophic.
Sure, there are environmental issues, false positives, renamed forms, etc., but that's the nature of having a legacy code base with few (if any) tests. In the long-run, if there's not a strong commitment to testing and solving these problems, then they'll remain. Ironically, they'll then be cited as the reason why testing isn't done. The problem is very circular in nature and, with all due respect, I've been testing for too many years to agree with your argument because with any significant codebase you can't safely refactor and then write tests (though refactoring will happen anyway and I do agree that if you have the tests after the code is written, that's better than refactoring with no tests).
Re:Hmm...
Mutant321 on 2008-07-21T10:59:36
Yeah, I think you're right, it's about a commitment to testing. Using integration tests is fine as an interim solution, but it must be used as a stepping stone to writing a real test suite (which to my mind is based primarily on unit tests). Otherwise you don't gain a lot.
Of course, it's easy for the PHB to say "we've already got tests, don't we?" But that's another issue, really.
You got me all fired up about writing tests for our user interface, in Perl even though the app's not in Perl
In fact, theoretically I should be able to figure out how to write a Perl test suite that logs into our application and executes many of its procedures remotely. Doing so would greatly increase my understanding of our application.
If you should happen to run across any material on programmatically driving a GUI app for testing purposes, please toss it my way!
Re:Alright!!
jarich on 2008-07-21T14:21:46
What's your operating system? If you're using Win32, have you looked at Win32::GUITest? This is a fine tool for programmatically driving a GUI app, and I _imagine_ that with careful work you could use it to do so for testing pretty much anything.
SweeperBot uses it just fine to play Mindsweeper.
;) All the best,
jarich
Re:Alright!!
Ovid on 2008-07-21T14:57:03
Holy crap. 6.5 Meg download just to automate Minesweeper!
:) Re:Alright!!
jplindstrom on 2008-07-21T15:31:20
How big is Java? The
.NET runtime? If you ship source, how big is the GCC download? 6.5M?
pjf on 2008-07-31T03:08:15
Of course, the 6.5 Meg download includes a Perl 5.8.8 run-time, Image::Magick, and a bunch of image capture DLLs, an unpacker, and all the goodies that otherwise come with an application built using PAR.
The end result, of course, is a user can download and double-click, and their machine plays minesweeper. They don't need to know anything about installing software, or Perl.
If you already have Perl and all the required dependencies installed, it's about 25k.
pjf@TinyGod ~/CVS/App-SweeperBot
$ wc bin/sweeperbot.pl lib/App/SweeperBot.pm
45 101 874 bin/sweeperbot.pl
918 3018 24167 lib/App/SweeperBot.pm
963 3119 25041 total
In all seriousness, App::SweeperBot contains a fair amount of code for reading the screen and doing mouse/keyboard controls. It may or may not be useful for other application automation/testing.
Re:Alright!!
jdavidb on 2008-07-21T17:33:39
Awesome!! My operating system is Fedora, but our project is Java, and intentionally so so that it can be cross platform. Half of our developers use Windows. (By choice, sigh...) And there've been rumors the past few months of new workstations for all, if only we'll run Windows on them. (Sigh...)
So I don't think it'd be too hard to get access to a place to try this out.