All the Perl that's Practical to Extract and Report

perl.org: peeling the onion

Robrt on 2004-08-13T04:55:35

If you've been following Planet Perl or Ask's Journal you'll know that we lost our main perl.org mailing list server (onion) on Monday when we installed really cool Cyclades AlterPath Power Management hardware. And then we had a big release at work, and couldn't get back to the data center to fix it.

Last night around 11:02PM, Ask started memtest86+ to see if that was the problem.

It wasn't.

This morning I went to the datacenter to try and fix it up. First goal was making it work under FreeBSD. Sadly, this didn't work. Basically, onion wouldn't re-stabilize under FreeBSD. It liked to spontaneously reboot. (And usually while fscking.) The FreeBSD 5.2 Rescue CD threw lots of errors. Single user mode was no better.

The only thing that I could keep running on the machine for more than 2 minutes was the (R)ecovery (I)s (P)ossible Linux rescue system. It booted. It let me ssh. It let me mount the FreeBSD partitions. It even let me run agetty so I could use our wonderful console server.

Using RIP, I was able to get all of the raw partitions over to our big disk machine, and from there we moved the files to the new mailserver. Luckily, I discovered that somewhere in our all 100Mb network was something limiting me 10Mb/s. This meant that the onion backup would have taken over a day. not good. A little bit of quick repatching, and we were back up to real speeds.

Anyway, As I write this, Ask is tweaking the configuration, I'm messing with postfix, and hopefully mail will start to flow in the next few hours.

Good work

Matts on 2004-08-13T07:47:26

Nice work guys. It really is a thankless task, and I think you and Ask do a storming job.

Re:Good work

barbie on 2004-08-13T10:21:34

I concur. Thanks for all your efforts guys.

Thank You!!

jbisbee on 2004-08-13T14:49:25

Thank you!! Thank you!! Thank you!!

Eye witness account

mattriffle on 2004-08-13T15:24:06

Thanks for all the hard work. :)

I got my subscription confirmation overnight from my attempt to subscribe to the qpsmtpd list a few days ago, so in my case mail is flowing again.

-Matt

Origin of onion?

VSarkiss on 2004-08-13T19:05:52

I take it the box was named "onion" because it brought tears to your eyes? ;-)

Re:Origin of onion?

ask on 2004-08-16T21:11:59

No, it was onion because of the 2nd State of the Onion. The first perl.org box I ran was called tmtowtdi.perl.org. :-)

Now most of the boxes are called exciting things like "x1", "x2" and so on.

  - ask