This summer just before I went away on holiday our IT support team upgraded the SAP WebMethods based Bussiness Connector Server that sits between our SAP system and my Apache/Perl front end. Everything seemed to be okay, and there were no problems reported while I was away, and even after I got back.
However all was not well. WebMethods had upgraded the system quite a bit converting the system from it's previous iso-8859-1 input/output to all new utf-8. As any Unicode aficionado will tell you the 7-bit ASCII characters in are also the same in iso-8859-1, and utf-8, so most things went through okay without a problem, however the top 128 characters in iso-8859-1 did not....
To fix things I had to re-jig the code, removing the iso to utf-8 conversions on the output stage, the XSL-T engine expected utf-8 anyway, and adding a iso to utf-8 stage on the POST to the business connector. All in all, it actually reduced the amount of code in there, so you can't complain too much.
The only problem I hit was that LWP when it POSTs data, uses the standard Perl length command to work out long the post is. Perl 5.6.x conveniently conveniently counts characters not bytes/octets, which means when the data is utf-8, it gets the wrong number of octets, truncating the file, and the BC Server's XML parser dies.
My quick hack was to count the length of the file before converting to utf-8, then in a use bytes; block count it again, adding the difference to the end of the file as spaces, which fools LWP. This was the solution I posted to PerlMonks. It's not very elegant, and I don't like it.
I then recalled that I had GHTTP on the box, and I thought I'd give that a try. Matt claims it's a lot faster than LWP, and on the whole I think he is right, but more importantly it was able to POST utf-8 encoded data without a problem, and I was able to fix my code, shorten it and speed it up at the same time.
Anyhow to the moral of the story. When the project was begun many moons ago a more senior Hacker asked if we had any tests, and I sheepishly admitted that we did not. If we had we would have been able to create a fix before the new BC server went into production, and we would have been able to do things at a more measured pace than the current head-long rush.....
