We had one of those "This can't possible fail, we didn't change code anywhere near there!" unit test failures today while doing a full test suite run after spending the day on a story card. The failing test file hadn't been touched for a while, nor had the code it was testing. But when I ran the test file by hand, it passed. Uh oh.
Long, hard experience has shown that 98% of the time this happens, there's some obscure interaction that's been missed or forgotten, or some other test just happens to leave the system in just the right (wrong) state to expose some wacky interaction.
Then there's the other 2%.
It turns out that our test harness uses file to determine how to run an individual test file. (Most of our tests are in perl scripts; a small few are something else.) And a nightly automatic update had just upgraded the database that file uses. Now, if 0x200 bytes into a file the characters "MP" appear, file identifies the file as some Apple block file thing, regardless of whether the file starts with #!/usr/bin/perl. And on this one particular test script, 512 bytes into the file...
