I have always wondered a bit about the mantra in the subject. Lately I have been exposed to problems demonstrating what it really means when you do not test the fragile parts of your application and it's environment.
I have for long wanted to write a developers handbook for my company and this is certainly going into the chapter on testing.
First problem was a web application, which was handed to me. The assignment was to get it into production. The application had been through UAT test and had been flagged as ok.
So I moved the application to the production environment, it took quite a few retries to get a few things playing, with a Build.PL which did not reflect all the requirements, pretty trivial stuff. The application was expected to talk to another application using Cache::Memcached.
So the two applications went on to each of there own servers and I started to look into getting them to talk to the production back-end via SSL.
The self-signed certificates I got did not look like any other certificates I could get my hands on so I had to get the back-end people to deliver some proper certificated.
Next problem was that I could not get Crypt::SSLeay to play along nicely with LWP::UserAgent. This is a known problem and reading the right documentation might have helped me earlier, but I was clueless at this time. I talked to a fellow perl-monger and was guided in the right direction. I did however file a request with the author of Crypt::SSLeay since I could not get my prototype to read my certificates, configured using HTTPS_CA_DIR and HTTPS_CA_FILE. Putting the complete path in HTTPS_CA_FILE did finally solve the problem, when I fellow Unix administrator got my prototype to fly. I back propagated his hacks and was enlightened.
I will post some more info on this when I get the time, since another fellow perl-monger took the time to go through the actual code.
Anyway I finally got the certificate part working and the applications started to work somewhat. Well I had to screw around with Memcached version on different FreeBSD machines - but I finally got it working, apart from one single thing. Which worked in the test environment, but apparently not in production. Namely the Cache::Memcached use, no parameters where simply transferred, just undef showed it's blank face.
After a lot of thinking, I attempted to serialize data using Data::Dumper and success data where transferred again. I have no idea why this was necessary all of a sudden, but I did not have the time to dig into the details, I was strugling to get the problems out of the way and I was running out of time, eventhough I was working late night after night.
So I got it working and after some testing we where able to go into production.
Way behind on my other project (talking daily to 4 different projects managers), I moved on with other things.
The lesson learned must be that you should attempt to test complex setups in a complex environment, do not just install it on a single server and expect it to work when it is distributed.
Thanks to the members of Copenhagen Perl Mongers who gave advice and where on IRC in the late hours.
