All the Perl that's Practical to Extract and Report

Encomiastic

Whammo on 2002-01-06T07:32:23

"Cold-hearted orb that rules the night
Removes the colours from our sight
Red is grey and yellow, white
But we decide which is right
And which is an illusion"
 -- The Moody Blues


I took another break from the Parrot Configure Mess that I'm creating to play around a little more with RC3 of Vorbis. It was mostly a trial run for when I recode my collection.



The alpha run was, for the most part, a burn factory. Whenever I'd sit down at my computer, I'd have two or three CDs being run through Grip - which was basically a graphical wrapper around cdparanoia and oggenc.



It was fairly automatic - filling in the blanks from FreeCDDB - and somewhat problematic - leaving quite a few Track Ones as headers only. In short, it pretty much did what I wanted it to - get the majority of my music onto the hard drive with as quickly and with as little hassle as possible.



Lessons learned:

  • GIGO has no better poster child than the CDDB services. If I'm going to have my information consistent and correct, I'll have to do it myself.
  • I need to take advantage of the paranoia portion of cdparanoia. Way too many defects survived the first round.
  • CDs are surprisingly variant in the volumes they're recorded at. I'll need to adjust the volumes accordingly.


The first, I'm not too concerned with getting right. Comments are simply name/value pairs, which I can update at any time with vorbiscomment. Since I want to put a database backend on the collection, I can fill one and export to the other. Since a lot of what I'm looking to index on isn't in CDDB anyway, I had already resigned myself to a lot of data entry.



The second, I played around with today. Basically, I'm running 'cdparanoia -wBX' and watching the smilies. As soon as I see how it reacts to various artifacts, I'll evaluate what else I should do. (I'm using WAV as the output format because it's very well supported by cdparanoia, sox, and oggenc, without having to mess with byte and bit orders.



The third is, by far, the most frustrating obstacle of the three, for two completely different reasons:



  • Although sox provides a facility for both determining ('sox {infile} -e stat -v') and adjusting ('sox -v {volume}') the relative volume of a track, it has no facilities for true normalizaton through gain companding. (It does has a vol module which adds a gain limiter parameter, allowing you to adjust low gains much higher than you could with the normal volume adjustment switch without clipping. It's still a linear adjustment, however, and the limiter only kicks in when clipping would occur. That has the unpleasant side-effect of making everything sound like a TV commericial, and does nothing for wanting to adjust the dynamic range of tracks whose peak volumes are nominal.) So, for that, I'm probably going to have to write it myself. Gain expansion is trivial. Gain compression, though, is not. Poorly handled silence detection and generation sticks out like a sore thumb.
  • Secondly, how do you determine what samples are your baseline for determining what gain adjustments you want to make? I count three different types of CDs that I'll need to consider. The first would be a run-of-the-mill album with distinct individual tracks, such as Journey's Greatest Hits. Each track could be normalized individually without disrupting continuity. The second would be an album with cross-track continuity, like Mike Oldfield's Tubular Bells 2. Conjoined tracks have to be normalized as a single entity. The third would be an album with conceptual cross-track continuity. And those are the one's that are difficult to determine. Holst's The Planets - do you normalize Neptune, the Mystic by itself, or as compared to Mars, the Bringer of War? Big difference.


The reference album I chose was the Moody Blues' Days of Future Passed - one of those that fall into that third category. It's got a large dynamic range, a variety of audio styles (percussive and padded instruments, as well as speech), and was an AAD conversion to CD, so that analog artifacts were forever digitally preserved. (The pychoaccoustical modeling techniques have a tendency to amplify them.)



For what it's worth, I elected to adjust volumes on the tracks individually, rather than on the album as a collective entity. The purists will argue that it destroys the vision of the album - Lunch Break and The Afternoon held the nominal peak volumes, so the album was already normalized as a whole. But I had always felt that Peak Hour was obnoxiously loud following a rather quiet beginning to the album, and the other tracks weren't even close to peak.



That brought me to the final test - the next version of the vorbis libraries and tools. RC2 (oggenc 0.8) really only supported 128 kbps (mean), so I wanted to stress both limits, as well as compare RC2 and RC3 (oggenc 0.9) directly. The RC3 generated file at 128 kbps (mean) came out slightly larger than the previous incarnation - on average, about 1.5%. (Unlike MP3s, which predominantly try to encode at a fixed bitrate, Vorbis usually encodes at a variable bit rate centered around a targetted mean. The latest release allows you to set a minimum and a maximum bitrate to limit or elimitate the swings in the bitrate.) I also tried 256 kbps (mean). Then I noticed that they had added a 'quality' switch to their arguments (values between 0 and 10 inclusive), which were favored over the old-style bitrate arguments. So I did one each of those, too. ('-q 0' is 64 kbps (mean), '-q 10' is around 498 kbps (mean).)



A lot of critics dismiss compressed digital music. They've got the basic facts straight: CDs aren't able to capture the analog experience, and MP3s, et al, aren't able to capture CDs. And if I had the opportunity to be locked in a reference studio, I'd probably complain. But I'm not. When I'm listening to music, I'm either at work, with one ear on my coworkers, one ear on the phone, one ear on my system, and whatever's left over on the background music emanating from $10 headphones hanging around my neck. When I'm in my truck, I've got engine noise, road noise, and my own yells of frustration at all the idiots on the road. At home, I've got cars driving up and down the street, computer fans running, and a ten-year-old set of cheap Pioneer speakers trying to be heard over the dishwasher. In most of these cases, the degradation of overall quality is affected more by the listening environment than by the lossiness of a particular format. For tonight's test, my "reference system" was a pair of digital Cambridge Soundworks speakers and a subwoofer running out of a SoundBlaster Live! card. I've a space heater and five computers running. The occasional plane flies overhead. In otherwords, every day listening. Music for the masses. A fanfare for the common man.



There were no discernable differences between RC2-128, RC3-128, and RC3-256. There was a slight, but noticeable, improvement with RC3-498. There was a slight, but noticeable, degradation with RC3-64. In both of the extremes, the differences were simply clarity. It was most noticeable during the recitation at the end of The Night. At the lowest quality, many of the softer sounds were muddled around the edges. At the highest level, they were as crisp as the raw audio. But unlike my big grief with MP3s, the intonation of the audio didn't change. MP3s seem to gain depth to the sound, particularly in the treble, around 160 kbps. At lower bitrates, the music just sounds flat. Vorbis at 64 kbps (mean), although a little fuzzy at times, still felt the same as at 498 kbps (mean). And that's damn impressive.



I'm tempted to just burn the space and go with the highest quality. But that means any streaming I do will need to go through a serious downshifting before heading out. Ices was hogging my CPU streaming at the native bitrate - I'd hate to think how much more it would need to actually do some work.



italics

pudge on 2002-01-06T15:39:45

Looks like you forgot to turn off italics, the whole post is italics. It's hard to read on my browser. :-)

Re:italics

Whammo on 2002-01-06T15:47:22

Oh, crap. Konqueror doesn't render them. I summon Mozilla!

Re:italics

Whammo on 2002-01-06T15:51:40

More accurately, Konqueror does render them, just not correctly, I guess. It stopped displaying italics when I closed the blockquote tag. Fixed.

Critics

chaoticset on 2002-01-06T21:07:48

Critics are given to criticism; the rest of the world, for the most part, just keeps doing what it did anyway.

OGG

ct on 2002-01-07T00:20:02

Very high on my list of things to do this year is redo my mp3 collection, currently standing at 519 discs at about 25gig.

A large portion of those mp3's are from years ago when I was encoding on windows with an old Xing encoder under windows, because LAME didn't exist. That particular version of the encoder had a nasty artifact in which any sound which faded from left to right or vice versa would Pop across the midline very noticeably. I've put up with it because I've never had enough tuits to do anything about it.

A while back a friend gave me a hint that has made all the difference in the ripping of my CD's. I had been using a Ricoh IDE CDRW or a no-name IDE 20x CD. There were a few discs that flat out would not rip, and many that had all sorts of problems. The answer is simple. Plextor.

I invested in a Plextor UltraPlex Wide. A CD-ROM that lives on Ultra Wide SCSI. It eats through CD's like nobody's business. Even discs that my other drives couldn't get through, this one rips right through without a problem. (Oddly, we tried a SCSI Plextor CD-RW and it had just as hard a time. There's something magical about the Plextor CD-ROM only units.) At $130 they're a bargain. Now, it doesnt rip at the 40x the drive is rated at, grip reports that it rips about 7-8x.

I'm going to re rip everything here soon, and was considering ogg, but I have two problems.

1) I've got several discs that have been broken, chipped or otherwise scarred beyond repair. I lost about 100 CD's that I simply cannot find after moving twice in 13 months. So I have alot of albums that only exist in mp3 form. From what I understand, mp3->ogg transcoding is a pretty dicey proposition, and since I stream alot, I'd like to have everyone in the same format.

2) Re-ripping is an astounding amount of work and there's a part of me that's a bit hesitant to do something in a software that's below 1.0. Yes, it's probably just an irrational reliance on an otherwise arbitrary numbering scale, but I know I'm not the only one who is shuffling around waiting for something called 1.0.

I could honestly care less about disk space, within reason. 120gig IDE drives are available for $208. I'll probably end up experimenting with ogg here shortly, but I'm really hesitant.