I often attend other (non-Perl) user groups in New York City -- mostly Linux and BSD groups. From time to time, when I identify myself as a Perl programmer, I am asked -- or, should I say, I am taunted with -- the question, "How is Perl 6 coming along?"
Now, up until two years ago, I simply ignored that question, because I had my hands full learning to become a good Perl 5 programmer and had no time/tuits to follow anything related to Perl 6. But since I was recruited to the Parrot project in November 2006, I've had to take that question more seriously. My stock response goes like this:I don't have the time to follow developments in the Perl 6 language specification. I do follow the development of the Parrot virtual machine. We expect to have our 1.0 release during the first half of 2009. You can view the Parrot roadmap.Of course, that's not the response that the questioner was looking for. So what I would like to know is if there is one site to which I can direct such questioners which says:
1. How complete is the development of the Perl 6 language specification. 2. What percentage of the completed Perl 6 language specification has been implemented on Parrot (Rakudo)?If we don't have such a site, what would be the obstacles to developing one? Thank you very much.
There have been many attempts to put such a thing together, but all aimed at different audiences, and they never have been maintained so far as I know.
I didn't reply before because I didn't have to contribute what you're looking for, but I was pointed at this again today and you've only got one reply... so... here goes.
The Exegesis and so on are pretty good. It's hard to say how complete they are until people have been using an implementation of it for a while. That's the nature of the beast. You don't know how much further until you get there, and in this case, that's a function of observing people use what you've made for a while.
The (I forgot what it's called... I blogged about it a while ago in a kind of state of things in Perl 5 land on my use.perl.org blog... the spec gimmie5 is built on top of) functional spec for Perl 6 is likewise good but not complete.
Then there's Rakudo.
This draws on an interesting topic... people looked at Perl 5 and said, "We can do better". That's the wrong kind of hubris. It's naive to think that you can make a 5.12 or 5.14 without a lot of prototyping ideas in modules, testing implementations in core, etc. Bad things happen when you try to pull a large, complex system out of nothing, whole cloth. Things like Second System Syndrome. If you haven't, stop and read The Mythical Man Month. Nothing I could possibly hope to offer as a reply here could begin to compete with what's already been written in that book.
I like Pugs because it tried to protype these ideas and let people play with it *first* then build a serious implementation through a bootstrapping process (translating Haskell to the Perl 6 that Pugs runs) second. A lot of people liked that idea. No one helped Audrey. Or few people did. I wanted to. Like everyone, I had excuses... not lasting more than four months at any job, trying to get consulting going to compensate, dealing with all of the fighting of an unhappy relationship. You get the idea.
Then there's Parrot. In general, I think it's a good idea not to complain about the people doing the work... lead, follow, or get out of the way... if you don't have anything nice to say... but again... there's a big gap of silence here. Maybe that silence speaks more than anything I could say. But I'm not known for my good judgement. See previous paragraph.
Parrot has a combination explosion of methods for accessing data (through array index, hash value, scalar, etc), what's being accessed, and which operator is being performed. Each PMC has a huge list of tiny accessors to accommodate these permutations. The idea was to make this fast, but this is a textbook case of a guy fresh out of college, full of self confidence, but badly lacking in real world large project experience. It's a good idea on the surface that's really a bad -- it's premature optimization. It made early prototypes go very quickly thanks to very short code paths. But later versions are finding these benefits completely offset by code bloat. Having too far unrolled the loops, the cache is being busted. Inner stuff, when Parrot is finished, won't fit in cache. Perl 5 has a runops and pp_hot.c that fits nicely in a moderately sized cache.
But that's not it... almost every aspect of Parrot follows this guiding principle of premature optimization. They decided on very tiny operations, sort of like the JVM (which should have been a clue that something was amiss -- how long and how much technology did it get the JVM running even passably?). Then, for *years*, the Parrot compiler guys went back and forth on call convensions -- which registers should be used for what when when a call is made from one function to another. They're in an early stage of development, completely lacking insight of a finished product, and they're spending *years* optimizing based on speculation and intuition -- something that most VMs never even think about. Even the JVM isn't that low level. By the time they're done, the LLVM is out and does a good job of this and the problem is obsolete.
Let me delve into a story here. MUD -- multi user dungeon -- was extremely formative for me. I spent ten years there. I watched MUD after MUD go down because they had too many players, the machine couldn't handle it, they needed to rewrite the *entire* codebase from 2.4.5 compat to 3.x compat to make it faster, and it'll let them make the game *better* at the same time, since they're already rewritting it, right? One by one I watched dozens -- almost every single one of the best MUDs go down for rewrites -- and never come back up. A few came back up... but one or two out of dozens. These were huge active places with dozens of programmers each. The MUDs that are still up and still popular either went through several false starts in rewrites and wound up something completely different than what they were, or more likely, never went down for the rewrite in the first place, and are *still* 2.4.5 compat. But here's the kicker -- the games that *did* come backup, did so *far*, *far* after CPUs became cheaper and faster and the benefit of rewriting was completely gone. 12mhz DEC 3100s were replaced by 180mhz SuperSparcs in that time period. Long after the fact, most of the new ideas in the 3.x driver turned out to be pretty lame, as far as running a game goes -- shadows were like AOP, intercepting function calls. The security model might have made anal retentive wizards happy but did nothing for the experience of the players. Etc, etc. Everything that the 3.x authors thought was important, including more performance, turned out not to be. It was just sirens that lured programmers -- programmers that have the same programmer mindset as programmers do today -- the "I can fix this! I can anticipate what will be needed! I can take a huge system and do better! The neat stuff I'm thinking of will be neat when I code it!" false hubris.
Programming is *slow*. Take on a large project, the world changes while you work. Wait a while, and most of the things you wanted just kind of happen.
I worked a Qwest (formerly USWest) for a while. Many chores involved tn3720's into regional IBM 390 mainframes. Other tasks involved an old VB2 or 3 GUI app. Others were Web based on a Netscape Enterprise Server written in an app in C++ against an Oracle backend with JavaScript that required 4.5 -- nothing else would work. Still other tasks were done in a Java client-side app that talked to still another database. Every few years, the programmers working on the grand unified GUI layer over the mainframe would find themselves using horribly outdated technology and far from finished, and a decision would be made to start over with the latest technology. This happened for 20 years without any effort actually completing a GUI that did everything the mainframe screens did. And some of the more obscure screens were still modeled after the punch cards they replaced. I'm watching people do this with Perl 6 implementations right now, but in parallel rather than only in series.
Now let's talk about Rakudo. It's impressive that it's as far along as it is. But that's the dumbest way to bootstrap I've ever heard of. If you have a language, such as BCPL, you change the parser to accept your new language, such as B. Then you slowly start changing the source code of the BCPL compiler to include B as its deemed safe. If you can build a compiler with the compiler, then build a compiler with that compiler again, and have the two binaries be identical, chances are very good that the code generation and parsing are working. Then before long, you have B, rather than BCPL. It was in exactly this way that C was created from B as well as B from BCPL. Working from scratch is what proud college graduates do, and it's guaranteed to turn into a ten year research program. Every now and then one succeeds at something, but no one talks about the numerous failures, as they do when you're actually working out in the industry.
gimmie5 effectively fits this BCPL->B->C model, but the waters are muddled a bit by Perl 5 not being self hosting. Still, if Perl 5 is able to parse Perl 6, more and more of the Perl 5 that parses the Perl 6 can be moved into C. There are various other options -- options that are incremental and save optimization for last, if ever.
Also on the topic of optimization... C is still relevant today, but it's becoming less so for desktop GUI apps and it's almost gone for Web stuff. C is concerned with very low level instruction level optimization. Perl 5 is concerned with handling things such as scope, temps, lists of complex types, etc quickly. As time passes, one kind of optimization becomes assumed while others become interesting battle grounds. Parrot is still fighting C's battles.
The whole idea of simulating a register bank in memory in Parrot is... well, let's say Parrot were a pure stack machine. Then the call convention debates would be almost non-existent. And someone could later come along and write a translator from the stack-based stuff to register file stuff, probably at the same time they were doing a JIT that translated to x86 or whatever. In fact, that would be the normal thing to do in that case -- use the x86's own register file. But perhaps they decided not to go down to x86 but still do a register file. Then the whole process happens later, nothing is built on top of it or assuming it, it's implementation can change without breaking anything or upsetting other designs. It's win-win. But by adopting this optimization premature, everything is delayed, code is fragile, everything has to change, etc. This is an implementation detail that should never have been exposed outside of some optimizer's own playground. And it may not even prove to be much of an optimization. Terribly hubris supposed it to be to the point where it was willing to make this gamble to build everything on top of it.
The second system syndrome exists partly in the Perl 6 spec -- everyone saw that coming. Everyone fought it while at the same time proposing stuff, myself included. Larry originally had a very small window of submissions and did a good job of keeping people posted as he sorted and scored the entries, and commented on them. But the real Second System Syndrome stuff is kicking in in the implementations.
I think the first lesson you learn out of college is that it is simply not a good idea to try to build large systems. The value plummets in relation to size. Value lies in the "quick hack". And if you can do a quick hack that grows, awesome. That's the best ever. But setting out to build a cathedral is just dumb.
In short, even if Parrot flies, it's going to be a bigger pile of crap than Perl 5, but the propensity for premature optimization will continue to sap time and effort from the things that matter -- as in the MUD analogy, from actual gameplay.
As for myself, I have some projects going, consulting still isn't where I'd like it to be so I'm living on ramen, but I'm toying with various quick hacks that might tie together different efforts rather than create new redundant efforts that further fragment interest, resources, and the attention of others who would like to jump in to this fray but get lost in the myriad of projects.
-scott
Re:Perl 6, Parrot, Rakudo.
chromatic on 2009-01-08T09:49:07
[People] looked at Perl 5 and said, "We can do better". That's the wrong kind of hubris.
Anyone who's patched Perl 5 and hasn't thought "We can do better" has insufficient imagination.
No one helped Audrey. Or few people did.
I tried. Pugs was unusable. This approach did not work. Note that for any perceived or real flaws in the Rakudo approach, it makes visible progress toward its goals every week, and visible, dramatic progress in every stable monthly release.
Perl 5 has a runops and pp_hot.c that fits nicely in a moderately sized cache.
It's funny that you mention this right before talking about optimizing based on speculation and intuition. To my knowledge, no one's profiled pp_hot.c to see where the functions end up and whether the PP codes in that file are actually the hotspots in modern programs. At least, no one's profiled real programs that way in years. (If you really want to make sure all of this code fits in the cache, figure out some way to pull all of the magic-checking code out of individual PP codes, so they can be substantially smaller.)
By the time they're done, the LLVM is out and does a good job of this and the problem is obsolete.
Y'know, I've known Chris Lattner since his undergraduate days, before he had the idea of LLVM, and certainly before he'd given up on GCC. Some of the Parrot hackers do keep up on the state of the art, at least enough to say that LLVM and Parrot have very different goals.
It's impressive that [Rakudo] as far along as it is.... But that's the dumbest way to bootstrap I've ever heard of.
Rakudo's never had VM bootstrapping as a goal. Thus I'm not sure why it's an interesting criticism that it handles VM bootstrapping poorly.
Still, if Perl 5 is able to parse Perl 6, more and more of the Perl 5 that parses the Perl 6 can be moved into C. There are various other options -- options that are incremental and save optimization for last, if ever.
Before Pugs went on hiatus, it took a work day for me to run the complete test suite. Sometimes you can't ignore optimizations.
gimmie5 effectively fits this BCPL->B->C model, but the waters are muddled a bit by Perl 5 not being self hosting.
Sure, and all you have to do to make control flow or laziness or iterators or parallelizing hyperoperators work in a Perl 6 bootstrapped on Perl 5 is either rewrite Perl 5's control flow from runops upwards or build a large chunk of Parrot in Perl 5.
I once wrote a patch to make resumable exceptions work in Perl 5. One reason it's not in Perl 5.10 or 5.11 is that making it work reliably with Perl 5 scopes and their C-level destruction of temps allocated within the optree itself is very, very difficult. It's not impossible, but the practical possibility of making it work requires refactoring a lot of Perl 5's internals, which probably means breaking a lot of CPAN and DarkPAN modules, because Perl 5's internals have never had encapsulation.
Add laziness to my hubris for that.
[Let's] say Parrot were a pure stack machine. Then the call convention debates would be almost non-existent.
Except for optional parameters, multi dispatch, named parameters... but flat/slurpy parameters are fairly easy. Did you know that LLVM's a register machine too?
In short, even if Parrot flies, it's going to be a bigger pile of crap than Perl 5....
Let's make a gentlemen's wager. I dare you to add a new feature to Perl 5 and the same feature to Rakudo. Which is easier? (I've done both. Does that mean I'm poisoning the well even bringing up this idea?)
Don't misunderstand. There are plenty of parts of Parrot I don't like -- and the Piethon contest was a mistake which hurt the project. I don't think your criticism are accurate, though.
Re:Perl 6, Parrot, Rakudo.
pmichaud on 2009-01-15T14:17:40
Now let's talk about Rakudo. [...] If you have a language, such as BCPL, you change the parser to accept your new language, such as B. Then you slowly start changing the source code of the BCPL compiler to include B as its deemed safe.
If this is the approach you advocate, then your comment is more properly a criticism of Perl 6, not of Rakudo. I'll note that when Larry wrote his parser for Perl 6, he didn't start by modifying an existing one either. I suspect that's because there weren't any parsers in existence that could be reasonably modified to parse Perl 6, which was effectively the state of things when Rakudo started.
It was in exactly this way that C was created from B as well as B from BCPL.
This is more properly a criticism of Perl 6 design, not of Rakudo's implementation of it.
gimmie5 effectively fits this BCPL->B->C model, but the waters are muddled a bit by Perl 5 not being self hosting.
Actually, gimme5 doesn't really fit the model you describe, because it doesn't start by modifying an existing parser or language. gimme5 is, in fact, a completely new parser built on top of an existing language. This is exactly the same approach that PGE and Rakudo have taken except that PGE was created using PIR instead of Perl 5. (And the decision to write PGE in PIR was based in large part on the advice of very wise, knowledgeable, and experienced language designers who recommended not attempting to use what had already been done in Perl 5.)
Ultimately, if you want to promote the BCPL->B->C model of language development, that's fine, but I think the language you end up with by using that model isn't going to be Perl 6.
Pm
Re:Perl 6, Parrot, Rakudo.
scrottie on 2009-01-15T19:06:21
I'm kind of sad to see you reply. I like to think that my kind of prattling on is completely off the radar of the people actually doing the work.
It's true that Perl 1 kind of came out of no-where. I don't know the early-early history here. Did Larry peek at awk's grammar and copy bits from it? I don't know.
I also know the Perl 5 parser is a bit horrifying and the muckiness of all of that is a lot of the catalyst for Perl 6, but, in theory, one approach to writing a Perl 6 grammar would be to hack up Perl 5. I'm not exactly advocating that. I'd be very pleased if that approach were feasible though. In my perfect world, Perl 5 is written in Perl 5 and has several good backends, sort of like HaXe does, and how pugs is trying to, but also like LLVM does. That is, backends targeting JavaScript, the JVM, x86 ABI, etc, etc.
I accidentally over-polarized the matter. The language written in itself is an ideal. It's not all or nothing -- just one pole. Writing the VM from scratch, programming an intermediate language in that assembly, writing another language in that language, etc, is the opposite pole. And that's the point I should have tried to make. And that's not a criticism of Perl 6 so much as of Parrot, at least in my mind. Pugs leverages other tools and had the goal of bootstrapping first then re-writing the language in itself, similar to the process by which B became C (but obviously not identical). So, according to this one criteria, it's possible to do better. That's not to say that there aren't other criteria and other ideals.
Hmm. While I have your attention, how closely is Raduko tied to PIL? I'm unclear on some of this -- there's an "not quite Perl 6" language written in PIL, and Raduko is written in that? I'm wondering how feasible it would be to target another platform for Raduko...
Regards,
-scottRe:Perl 6, Parrot, Rakudo.
Aristotle on 2009-01-19T22:26:11
I think you overestimate the difficulty of writing a parser for a language and greatly underestimate the difficulty of equipping a two-decades-old runtime with evaluation semantics like laziness that it was never designed for. The BCPLBC example is somewhat of a red herring in this because all of the languages are in the “portable assembly” class largely defined by ALGOL, without significant differences in evaluation semantics or memory model.
I made a very small proposal for 5.12 and it took a day of sleuthing by perlguts gurus to actually implement and then unkink all the edge cases that curled up. I don’t think it can be argued seriously that perl is anything less than hard to hack on.
In contrast, well… just read the Rakudo source. Since the docs and written spec aren’t always congruent right now, I did a lot of source diving when trying out things – unlike Perl 5 interpreter, this is actually a feasible endeavour for absolute novices.
Re:Perl 6, Parrot, Rakudo.
scrottie on 2009-01-20T03:08:22
I'm confused. I said that the ideal case of morphing one language into another did not apply to Perl 5 and acknowledged that one of the primary reasons for a rewrite was the Perl 5 core.
Can you double check that I said what you thought I did?
-scott
