Is it just me, or is everyone writing an HTTP server these days? After Apache 2 became solid, it looked like there wasn't much of interest left to do in the world of HTTP servers, and the field had been fully commodified. CPAN had a half dozen or so Perl HTTP servers, all of which were fine for entertainment but not useful for real sites. You'd hear some crank on Slashdot shouting about thttpd (I swear that wasn't me), but it didn't set the world on fire.
Then the single-threaded servers started showing up in earnest. A non-blocking I/O approach to networking is well-known to scale better than threads or processes, and it appealed to developers in a very primal way -- it's fast! Well, not so much fast, since you'd run out of bandwidth long before that mattered, but you could handle lots of open connections to slow clients without any trouble.
Lighttpd quickly became a star, especially in the PHP and Ruby worlds. (Why were Rails developers looking for a faster web server rather than trying to fix Ruby's performance problems? Probably because it's a much easier problem.)
Somewhere in there, Perlbal made the scene. It's a bit of a hodgepodge of features, having been developed to suit some particular in-house project needs, but an interesting sort of glue project to fill gaps in the Perl web app deployment story.
Some of the Rails guys then decided they didn't like FastCGI and would write their own HTTP server to replace it, called Mongrel. So far, the benchmarks I've seen make it look like performance has gotten worse compared to what they had with FastCGI, but it's still early so maybe they will improve that. They say they were doing it because the FastCGI implementations all had bugs, so maybe they don't care if it's slower anyway.
Meanwhile, people started popping up on the mod_perl list saying that they had built their own single-threaded servers. I usually ask people two things when they say this:
The only good answer I've heard to the first question so far is to ship the blocking stuff off to some separate persistent processes (e.g. mod_perl, PPerl, etc.) that you talk to over non-blocking I/O, and pick up the results when it's done. This is what Stas Bekman did with the single-threaded server he works on at MailChannels (for blocking spam). It's also what Matt Sergeant seems to be planning for his new single-threaded AxKit2 HTTP server.
Meanwhile, back at the Apache 2 camp, mod_proxy has picked up useful new features like basic load balancing and people are experimenting with hybrid threaded/non-blocking I/O process models.
It's good to see innovation happening. Sometimes I do wonder if people are chasing the right things. I find it pretty easy to make a screamingly fast web app with basic Apache and mod_perl these days, so maybe pushing things in a direction that makes development harder (as I think single-threaded programming will be for most people) is not the best move for all of us. High-performance has an undeniable allure though, especially for people like us who still have to convince managers that Perl is fast enough for a web site. (Duh. Maybe you've heard of Amazon?) I'll certainly be paying attention though, to see what Matt and everyone else cooks up.
I'm currently using it to develop a debugger (very prelim view), hopefully that will shake out HTTP::D::T a bit more.
BTW: does anyone know the status of Servlet ? It hasn't been touched in 5 years, and I can't figure out what sort of HTTP server it uses...in fact, I didn't even know about it until I was searching for other HTTP::Daemon extensions (rather poor toplevel name choice...)
Re:Multithreaded Perl server...
perrin on 2006-09-01T15:58:10
I think Brian Mosely has left the building. He used to be on the mod_perl list, but we haven't heard from him in years. I'd steer clear of Servlet at this point.
Re: HTTP Server Fever
perrin on 2006-09-01T21:10:47
The Ruby people pushing for Mongrel were fleeing Lighttpd and FastCGI, which they claim is buggy.
to your plugin, which is just so sweet compared to mod_perl2 config directives. Similarly hooks are easier to write.sub conf_Foo_Bar;
Re:Some justification
perrin on 2006-09-01T21:31:35
Adding config stuff was pretty bad in mod_perl 1. It's supposedly easy in mod_perl 2, but I've never wanted to add a config directive so it's not something I pay attention to. I really don't use any of the Perl config stuff in mod_perl at all. It just doesn't appeal to me.
Profiling, quick restarts -- well, they work for me in mod_perl. (This reminds me of working at a place that used IBM WebSphere, where restarting the dev tools to pick up a code change took 10 minutes! What a pig that thing was.)
You might be able to get the best of both worlds on the auth front by using AxKit2 as your application server and proxying from an apache2 front-end that runs an auth plugin. I do this with mod_auth_tkt, which lets me keep the actual auth code in Perl but have the proxy do the auth checks on static files. You'll lose the simplicity of your setup though.
Re:Some justification
Matts on 2006-09-01T21:55:19
I think partly people don't add custom config in mod_perl apps because they're hard (still non-trivial in mod_perl2 - that was one of my stumbling blocks in doing a straight AxKit port to mp2), and partly because they are satisfied with "PerlSetVar" which I never really liked as it didn't seem to cope with multivalues terribly well, and partly because none of the introductory docs cover them. I've tried to make sure my introductory docs jump right into them because they're terribly useful.
Another option I'm thinking about is if I can emulate the mod_perl API enough then I might be able to make some (but probably not all) auth plugins just drop right in.
But that's for another release a bit later I think:-)
As for "Profiling, quick restarts -- well, they work for me in mod_perl" -- I have to be fair that I haven't used mod_perl 2 much, but I still find the whole apache startup time to be way too slow. With AxKit2::Test we can hook right into the startup process with a pipe() so that we know when it has started (and if it has failed) a lot faster, meaning the tests get going a lot faster.
Regarding profiling: I'm not sure whether it's better or not, but under AxKit2 you know exactly how much of your time is spent processing the request/response as well as is spent in your application code. Obviously you trust Apache/mod_perl2 to do that very quickly so it's not a huge issue for you, but I kind of like the fact that all that info is available to me.
I'd also urge you to play with it a bit - you might actually find you like it:-)
