Today I was asked about scaling mod_perl to handle really big loads. They mentioned the possible direction of a J2EE solution.
Now I didn't have any statistics handy to back why mod_perl can scale, but I after mentioning that mod_perl runs Slashdot.org, and TicketMaster.com, I made a bold statment:
"I'd wager my career that mod_perl can scale as well as any other technology."
The response?
[laughter]"Would you put that in writing?"[/laughter]
In this journal entry I hereby make good on my answer of yes to that question :)
we've had no problems scaling perl
TeeJay on 2005-09-28T09:30:46
..in fact there is no reason you should - the hard part is designing a scalable system rather than the language you choose.
the only times that the choice of language or platform has been a problem as been with some ASP applications I have had the ill luck to work on (not design, or develop I hasten to add)
A talk you really really want to find
Alias on 2005-09-28T12:36:11
Someone from Yahoo was at OSCON, giving a talk called "How to serve a billion pages a day with Perl" or something similar to that.
He gave a great overview of all the different methods you can use to scale the application. Which ones work, which ones don't. And which order to persue them in.
... pause...
actually, I managed to find the overview
When: 7:30 PM, Wednesday, July 20th, 2005
Where: ActiveState/Sophos, 580 Granville Street, Vancouver, BC
Speaker: Kevin Scaldeferri
Topic: How to Serve a Billion Requests a Day with Perl
This presentation discusses the challenges of building complex services
that can handle very high query rates with low latency and high
availability. Overture, the sponsored search division of Yahoo!, has
developed a number of systems that have successfully scaled over several
orders of magnitude in both data volume and query rate, and Scaldeferri
will discuss the ideas behind some of these. General topics covered will
include:
* Caching using local and distributed caches
* When and how to distribute your application across multiple
machines
* Use of asynchronous communication techniques to increase
throughput and ensure acceptable response times
Scaldeferri also discusses the interplay between different levels of
abstraction, from high-level business logic, to practical Perl
programming, to low-level OS and hardware behavior, and how all of these
affect the design and architecture of such systems.
Bio: Kevin Scaldeferri is a lead developer at Yahoo!'s sponsored search
division, Overture, where he has spent the last three years taming an
ever-growing Perl code base that currently serves over one billion web
requests a day. His professional interests include programming languages
and software development practices. Outside work, he enjoys cooking,
reading, modern art, and "challenging" music.
Re:A talk you really really want to find
Phred on 2005-09-29T00:03:15
I was unable to attend OSCON this year but when I looked at the speaker summaries, this was one of the talks that jumped out at me. I'd really like to attend it at some point, or peruse a written copy. Thanks for the tip on that.
agree
Qiang on 2005-09-28T16:56:32
i agree with TeeJay. it's the part designing a scalable system more important than what language you choose.
Brad Fitzpatrick from livejournal gave a good talk on how they run a successful website. It's not only about mysql but perl too.
http://www.mysqluc.com/cs/mysqluc2005/view/e_sess/6257talk available to download at the end of that page.
also, mod_perl website contains some nice doc on how to setup a scalable system with mod_perl.
Re:agree
Phred on 2005-09-28T23:51:34
I've implemented some of the open source LiveJournal tools such as MogileFS and Perlbal, and they've been working great at my current project. That combined with parts of the Practical mod_perl book, the mod_perl website, and many other sources have given me the knowledge to scale it, and I've implemented those methodologies in a couple of places.
uh... Amazon?
perrin on 2005-09-28T18:38:45
It's FastCGI, not mod_perl, but it's a huge site that most people have heard of and they use Perl (Mason specifically).
Re:uh... Amazon?
Phred on 2005-09-29T00:00:28
/me smacks forehead in one of those "Why didn't I remember that at the time?" moments.
I dug into the mod_perl site docs last night and found this link of sites running mod_perl. The question was more in the context of using Perl as the technology of choice though, so FastCGI, Mason, etc. are all relevant here. I found this link also about HTML::Mason powered sites.
TicketMaster and Slashdot jumped to the front of my mind when the question was asked, but next time the question comes up I'll point them to this journal entry.
Re:uh... Amazon?
perrin on 2005-09-29T00:11:03
I wrote a whole talk on this subject, mentioning Amazon and others, which I gave at OSCON and ApacheCon a couple of years ago. I even have an article form of it written. The only reason I haven't given it to perl.com yet is that the section on queue systems is kind of weak. I was hoping to rewrite it when I put my queue system on CPAN, and since I haven't done that yet...
Re:
Aristotle on 2005-09-29T22:41:45
http://www.oreillynet.com/pub/wlg/7930
A more recent story.
