Like a ethnic dish, some mistakes we are destined to repeat.
In the course of my Linux experience, I've hosed many a server. Sometimes I've assigned swap space on an external RAID box. Sometimes I've created overlapping disk partitions on hard disks. Sometimes I can't even tell what I've done. This is what makes Linux and Unix fun.
However, there seems to be one mistake I can't stop making. A Unix system layered like a onion. Each layer depends on the preceeding one to work. At the lowest layer is the kernel, which futzes with the hardware and attempts to make all Linux/Unix systems appear to be the same from an application programmer's point of view. The bridge between the kernel and applications (and even basic system services like init and bash) is the libc library. Without this library in good working order, it becomes difficult to exit one's shell, let alone reboot the system cleanly.
Now, I upgrade various bits of my system all the time. I'm not afraid to recompile the kernel. I'll update the C compile and attendent utilities with wild abandon. However when it come to updating that most scared of files libc, my track record is not so hot. You see, get lulled into thinking that a package manager like rpm can Do The Right Thing to install libc without hosing the system. In fact, a google search relieve a testimonial from a user with the same version of RedHat I have successfully upgrading his libc to the version I also desired.
Alas, not all messages should be taken at face value.
There are three libc (really glibc) packages that needed to be successfully installed. The first two worked fine. The last failed and precipitated a cascading failure that resulting in my reformating the root partition and laying out a fresh install. I got to bed around 2A. Good times.
One of the last things I did was to install Spam Assassin, since I was already getting swamped with spam. Of course, sendmail tripped me up. I forgot to add symlinks in /etc/smrsh to procmail and spamassassin. So imagine the hilarity as I awoke to find 4000 messages being downloaded by fetchmail, nearly all of the bounces from my local sendmail saying "I don't do procmail, biznatch." Boundless joy!
With that sorted out, DNS back in place, and offloading DHCP to my linksys router, I'm pretty much back up and running with the essential services. Of course, I'm missing a lot of formally installed Perl modules. Like SOAP::Lite, which I need to post this journal.
Sigh.
Re:Try Debian
Dom2 on 2003-10-28T21:29:39
That's so Linux user. There's a little script that all Linux people go through when they're doing support:
- Doesn't work? Upgrade your kernel!
- Still doesn't work? Try a different distribution!
- What was your problem again?
I'm sorry. I've been reading jwz too long.
FWIW, I've been using RedHat a while and never had any problems with glibc upgrades screwing up. Lots of problems with locales (solution: nuke), but never glibc.
:-) -Dom
Re:Try Debian
autarch on 2003-10-28T22:26:22
Well, there are certainly other choices, like not using Linux. I doubt that any other Unix-y thing will be much better than Debian in terms of ease of upgrade, though.
Then there's Windows, which we all know and love or hate. As for Mac OS X, it seems nice, except for all the things I need to do my development work that only sort of work/compile properly there. I don't want to do a compiler dance every time I need some new Perl module with XS installed.
Basically I've really only used two Linux distros, RH and Debian. RH has always sucked, and Debian was wonderful from the beginning. It's not perfect, but it's better than RH.
[insert obligatory anacdote about Debian and its ability to update libc three times before breakfast here]
That out of the way, some advice. Back in the mists of time when Apple Powerbooks were black and named after streets in New York, some Linux nutters decided it would be cool if they could run something other than a crappy co-operative multitasking system (translation: we crash by mutual consent) on Apple hardware. Enter Debian Linux/PowerPC. I had interesting times in 1998 when they were just getting off the ground. There was no stable tree, just unstable and yes it was unstable. A libc upgrade causing catostrophic system failures was a regular occurance.
So what I learned to do, and my advice to you, is to do a static compile of your basic utilities to have in an emergency. fileutils, bash, textutils, rpm, etc... Or keep a copy of sash (Stand Alone SHell) around and set root to use that. It'll at least let you do *something* when libc fails.
Or you could just use Debian. *p0inT and l4f*
Re:Point and laugh at the funny Redhat users.
jhi on 2003-10-28T20:42:09
> So what I learned to do, and my advice to you, is to do a static compile of your basic utilities to have in an emergency. fileutils, bash, textutils, rpm, etc... Or keep a copy of sash (Stand Alone SHell) around and set root to use that. It'll at least let you do *something* when libc fails.
Or how about a static compile of Perl...?Re:Point and laugh at the funny Redhat users.
Dom2 on 2003-10-28T21:19:42
Hmmm, a static zsh probably does nearly as well these days. By the time you've got zsh/files, zsh/stat and zsh/zftp modules builtin, you're away...-Dom
Re:Point and laugh at the funny Redhat users.
schwern on 2003-11-03T00:31:03
> Or how about a static compile of Perl...?
Don't be silly, nobody uses Perl.;)
Moving from RedHat may be an option for home use, but my two jobs won't be migrating from RH any time soon. And here's a confession: I like Red Hat. Yup. I like RPMs and SRPMs. I've even figured out how to use chkconfig to register my sysV init scripts. Would it be cool is rpm was more like CPAN, in that it sought out missing dependencies over the net for you? Sure. I know that's what RHN is all about.
That said, only a fool wouldn't try to avoid making the same mistake again. I will certainly look into keeping a few stactically compiled apps around, although please note that even init was hosed by the upgrade (talk about a candidate for static compilation). I suppose I should compile some shell and halt at a minimim. Perhaps sync and mount (and umount) too. Then I can use the RH install disk to repair the system.
