I want to move to the next level in testing all my code, and writing tests before I code. One thing that's stopping me is I see some very difficult situations to test. How do I test something graphical? For example, I have routines that are generating charts and graphs in PNG format. How do I test these? How would I test a GUI interface for something? Also, how do I test routines with side effects? (i.e., routine prints to the screen rather than or in addition to returning a value).
I understand that writing my tests first and designing with testing in mind will go a long way toward making my code testable. I also understand that experience will go a long way toward teaching me to test anything. But so far I still just can't foresee how to do some of these things (or the ways I can imagine to do it are impractical).
Any sources people could point me to about more complicated testing like this, books, urls, periodicals, or forums, would be immensely appreciated. I need to see what path people took when they came this way before.
You have several options:
I don't do much graphics programming, but if you separate display from logic, you'll have plenty of opportunities to test the essential behavior.
Re:Break it Up
jdavidb on 2003-12-09T21:49:10
You can test the building one automatically and the printing one by hand if you need to.
Ugh, no testing by hand, thanks.
:) That's why the question "how do I test a routine that prints something" still exists after I design for testing and break the routine up. The other ideas were good, though; thanks.
Question: How do I test something graphical? For example, I have routines that are generating charts and graphs in PNG format.
Answer: One way would be to pre-generate "known good" charts and graphs, save them in a test_data directory and, run your routines and see if the output files are the same.
Question: How would I test a GUI interface for something?
Answer: I've struggled with this and for the most part, I don't think there are good, generic answers. One suggestion that I've heard is to have the system take screenshots of the GUIs and use the method listed in my previous answer. This is very fragile and non-portable, however. Further, a decent answer would vary depending upon how you create your GUI (Template Toolkit front ends would be tested much differently from a Tk framework).
Question: How do I test routines with side effects? (i.e., routine prints to the screen rather than or in addition to returning a value).
Answer: one way is to build hooks into your system:
# somewhere in package Foo:
$self->_print($some_text);
And in your test suite:
{
my $message;
local *Foo::_print = sub { $message = shift };
my $result = $object->method;
ok($result, 'Calling method() should have a result');
like($message, qr/$some_text/, '... and it should also print some stuff');
}
Not only does it make it easy to trap the data, you later have a very easy way of logging, trapping, or munging those _print messages.
As for the last technique, I used to hate to do that because it seemed "wrong" to build testing hooks into the code. As times goes on, though, I've found that it makes testing easier and I've never found an actual drawback.
quick and dirty GUI testing
gizmo_mathboy on 2003-12-09T21:59:10
What we did at an old job was get a known good image (chart, graph, or even a snapshot of the desktop).
Then create a checksum (md5 is kind of small but I liked to use sha1). Then you just need to create a checksum on the test image and compare checksums, and then you could flag an image for human review.
There was some other super simple things I came up with as well for images and rough UI testing.
Re:Testing Strategies
ferrency on 2003-12-10T15:47:20
I've occasionally used another technique for testing the text output of legacy scripts to prevent regression while refactoring or fixing bugs. Redirect all output to a file, for both the new and old versions of the script, and then use "diff" to detect differences. This is certainly worse than fully automated testing, but much better than fully manual testing.
Redirecting "all output" is slightly more challenging if the legacy code doesn't have the hooks Ovid recommends, but not impossible. Although "print" is usually called frequently, "open" is typically called infrequently. Adding a conditional to each "open" which causes output to go to STDOUT or STDERR instead of a file/pipe when in test mode is not too hard, and takes care of most output.
This just leaves bidirectional pipes and sockets, which are much more challenging. For these, connect to a mock server/system if possible. But it is not always possible to build an accurate mock system if you have incomplete knowledge of the system you are connecting to: if you have the ability to test the validity of the mock system, you don't need the mock system in this case.
Image output can be considered generic data output for testing purposes, if you have complete control over the input data, and the output format does not change at all.
GUI testing is a completely different problem. For web-based systems, functionality of the GUI can be tested automatically, but it's not always easy.
Alan
Do you mean test that the interface does what you think it will on the back end, or test that the interface is usable?
