I figured out my problem, and if I hadn't spent the whole last hour chasing it down, I'd be less happy about figuring it out. :)
I realized I was looking in the wrong spot. My original code was probably spot-on in the place I'd been changing and printing and debugging madly; it was waaaay the hell back, at the original sub call, that I managed to turn two arrays into one and generate a hell of a puzzle for myself. (Yet another reason, if you're only calling a sub once, to keep the sub call and the sub dec next to each other if it's humanly possible, I guess.)
Hopefully, I'll be more careful with refs from now on. Since starting from the original ref generation and moving through the code solved the problem of where to look, next time a ref issue pops up, I'll probably do that first, instead of trying to modify my derefs.