(I wanted to title this journal entry "The relationship between a language and its toolchain, and why Perl 6 scares the hell out of me" but it didn't fit)
For the record, this is not an anti-Perl6 rant. It is a warning.
For language designers, one of the foundational concepts is how simple the grammar is going to be. Being a human interface device, languages go beyond math (where there is generally a single truth) and engineering (where there's generally a limited set of widely known best practices) and can get into the realm of personal preference and fashion with few clear guides for the "best" way to do something.
The reason this dimension is so important is that the language imposes limitations on the types of tools it is possible to write. Some types of tools simply are not possible to write at all if the grammar has certain features.
I hypothesize that we can break down the human "language experience" for a general-purpose language as being a combination of the language syntax itself, the tools for it, and the size/quality of available libraries. These are by no means all the success factors, but are the ones that in my opinion make up the influence on the individual user.
(I explicitly ignore languages with features like "proof-carrying" that are essential for specific domains like cryptography and make it worth the pain of learning Haskell or proof-carrying Ada)
According to my hypothesis, the danger here then is to add excessive expressiveness to a language with the intention of it being pro-user, at the cost of fatally crippling your toolchain and hurting the user more than the benefits gained at the language level.
The second risk here is what I'll call the "Personal Language Anti-Pattern". I first heard examples of this from some old-school Lisp hackers. The typical way of describing it goes something like this...
"I can write something in 4 days in Lisp that takes most people 20 in some other language. I just spend 3 days modifying Lisp to solve that type of problem, then 1 day solving the problem".
The anti-pattern here is that you end up going way beyond TMTOWTDI. You don't get two, or three, or five ways to do it. You end up with a different language for every single person and every single project.
Forget about maintaining projects written in crufty Perl. Imagine maintaining code where every single project is written in its own mini-language (although they all look a bit lispish).
In Perl 5 we achieved this depressing state with source filters, but mostly managed to keep it under control with culture. "Source filters == bad" is our cultural norm.
The risk with Perl 6 here is that the ability to safely modify the language is going to be taken as permission to modify it early and often.
I'm not talking obvious positives like "use physics;" here, I'm talking My::Project::Lang here, in the same vein as the "God Object" anti-pattern... Where I don't want to end up is "BioPerl... the Language!", where DNA sequences are a known literal (because it starts with a capitalised GTCA).
my $sequence = GATTACA;
This seems like a seductive option, but the cost is you throw away most of your developer tools.
While I had originally thought that "easier to parse" was a part of Perl 6, this has apparently been removed or was never what I thought it was.
What Perl 6 actually is is easier to IMPLEMENT.
That is to say, we won't be in the worst possible situation of having a dynamic grammar that can't be reimplemented at all, because there is no grammar beyond "what the implementation does".
So we will have A grammar, but now it is a grammar that is EXPLICITLY changable.
So consider this step 1 to toolchain bliss.
It's still removes the possibility to implement most useful tools.
So, as I see it anyways, here is the rest of the steps.
Step 2 - Deterministic
The key to the really awesome tools is that you need to have a way to READ the code, without necessarily having to RUN the code.
BEGIN blocks (and everything similar) really screw this up for us.
If you need to execute code to read code, then you need to execute arbitrary code in order to read arbitrary code. And right there the phrase "execute arbitrary code" should be more than enough to explain the problem.
In one hit it creates the limitation that you can only every create tools that run your OWN code, you can never write tools that run anyone else's code.
As an example of how this can hit in unexpected ways, in Perl 5 if you have Komodo (or anything else that does background linting) installed it's quite easy to create a totally innocent-looking link on a webpage that will delete your home directory.
So compile time string-eval has to go (and BEGIN blocks or anything else that does it). It's also the death of having a "sub import" that can do anything you like.
In exchange, you get code you KNOW is going to be parsable, if the entire program is valid. None of this BEGIN { exit if rand > 0.5 } stuff.
Since it is so trivial to implement (one line of code in PIL), for the next parrot release there should be an experimental --xdeterministic flag you can pass to perl6 to forbid compile-time execution.
None of this makes interesting tools more POSSIBLE, it just makes it safe to open up a project from a third party without wondering if it is going to install a root kit or not. Which goes a long way to creating the incentives to write the tools at all.
Step 3 - Finalized Grammar
The other ugly problem is in the idea of having a compile-time-morphing grammar AT ALL, and completely blows away the possibility of having a "PPI6" that is sufficiently complete to handle all documents (and along with it kills perlcritic6 and sqlinjectiondetector6 and perltidy6 and other stuff).
If the syntax and semantics of your document is not stable DURING the document being parsed, how are you supposed to generate any form of semantic model (ala method-name completion or SOAP API auto-generation) or even a syntactic document model (ala PPI).
You can't describe anything, simply because you have no idea what you will need to describe in advance.
Perl 5 is horrible in this respect because the grammar contains an "operator/operand" state that flips back and forth every other character (which is the underlying cause of the "/" problem, and 6 other characters).
But at least it's only a boolean flag, and I could fudge my way around the problem by heuristically guessing well enough.
Once grammar modification is easy, any reasonable likelihood of fudging goes out the window because it can change in so many more ways.
You end up with a language which is expressive as hell, but where the most sophisticated editor you can create is vi (with no syntax highlighting allowed).
Assuming such a thing can be created, my bet for the "use strict;" for Perl 6 would be something like "use v6-static;" which would guarantee the file sticks to the official primary grammar for that document and allow you to safely use source code analysis tools on the file.
This could even be useful without determinism, as it would at least let you prove that the compile-time code wouldn't modify the grammar, and so you could still safely model the source code without having to risk compiling and running the BEGIN block.
It would also mean you solve the other big stepping stone for tools, the ability to do useful things with a document BEFORE it is a legally correct program.
This covers everything from parsing "use Win32;" on Unix to "use Not::Written::Yet". The ideal for a parser is a context-free (only needs one file) symantic parser you can safely fuzz-test without it exploding.
PPI is merely a syntactic parser you can safely fuzz-test.
And be under no illusions that it can be replicated in Perl 6. It was only BARELY possible to implement it for Perl 5 and even I wasn't sure I was going to find a good enough path through all the impossible problems until a few months before it was finished.
Now if we are indeed walking into this trap, there's certainly ways to avoid it.
Removing determinism would be useful (and easy to implement) but possibly impractical as it removes the ability to do platform-adaptive code that picks dependencies at compile-time. Goodbye File::Spec...
Finalizing the grammar by default somehow is far more interesting in my opinion. It enables all syntactic and some of the semantic tools, and will discourage flippant grammar modifications.
If you REALLY need to do something exotic, you should be willing to pay the price for it by doing something like "no determinism;" or "no tools;" so that it's clear to anything parsing your code that it should stay the hell away unless it is either running the code, or willing to take the risk or exploding violently.
Or at least the tool is CONFIGURABLE to act safely, instead of being vulnerable to exploit by default. And before you mention Safe.pm, consider "BEGIN { while(1) { $_++ } }" or see a great talk by an Australian University lecturer that teaches a Perl course called "Safe isn't" in which she explores all the ways her students violate her computer from inside Safe containers.
Re:reusable toolchain?
rjbs on 2008-01-29T15:37:27
in the absense of any BEGIN { } declarations or importing of macros/grammars from other compilation units (this is something you can statically check for if the unit is already compiled)
I am probably being thick, but once they code is compiled, haven't you already run the BEGIN block and all its arbitrary contents?Re:reusable toolchain?
nothingmuch on 2008-01-29T17:15:18
The begin block itself has to be fully parsed before it's run.
Similarly a macro or grammar extension coming from another compilation unit has already been compiled.
At this point you can examine their code in a manner much like Safe does (existing problems in safe are an implementation issue, not a conceptual one), and run the code with some resource limitation if necessary (if this wasn't possible then we wouldn't have javascript;-)
Furthermore, if you deduced by static analysis that these blocks cannot affect your code (Because, for instance, they never address the grammar classes, or they don't touch any symbol table, they never run string eval, etc) you can deduce that this code will not affect parsing in any way, and thus skip it entirely when you are only parsing (as opposed to compiling).
On an unrelated note, another benefit of this extensible toolchain approach is that if you *do* implement a DSL, then any tool can at least still parse it properly, and if the DSL is very simple or the tool is easily extensible you can get the tool to treat this code correctly too.
Imagine PPI being able to correctly parse Switch.pm's statements because Switch defines it's magic in terms of a real parser extension. It produces a new type of tree node that PPI using tools might not know how to handle, but they can at the very least safely ignore them. If the node does roles (like a 'ControlBlock' node for a Switch statement) then the tools might even be able to work with them anyway (like Perl::Tidy fixing parenthesis for Switch statements just like it would for if { }).Re:reusable toolchain?
Alias on 2008-01-29T22:13:49
> At this point you can examine their code in a manner much like Safe does (existing problems in safe are an implementation issue, not a conceptual one), and run the code with some resource limitation if necessary (if this wasn't possible then we wouldn't have javascript;-)
You don't need to run Javascript in order to parse it, since it has (I think) a static syntax.
Also, this comes down to practicality.
"What percentage of CPAN can this parsing strategy handle?"
As a context-free document parser, PPI can handle as close to 100% as makes no difference.
Remember import methods are BEGIN-time code here too... so ANY "use Foo" kills you.
Anything where you don't have dependencies installed kills you.
Anything where you read config files at BEGIN-time kills you.
Everyone doing things like BEGIN { %foo = map { $_ => 1 } qw{... } } kills you.
If you were to work out what percentage of CPAN you could support under a "just use the native parser" approach, I think you'd find it is a small percentage indeed.
Re:reusable toolchain?
nothingmuch on 2008-01-29T23:31:18
With respect to dependencies that will indeed fail to work, but for use Foo that's untrue - Perl 6's importing semantics will support real linkage of symbols for the benefit of compilation units. The method 'import' and glob assignments are not supposed to be the only way to actually import symbols anymore. This solves a lot of issues.
As for reading files etc in BEGIN - that's also handled differently - there is no guarantee that a BEGIN block will run every single time you run the program, it is fair game to save the output as constants in the compilation unit. Reading configuration files is supposed to happen at INIT/CHECK, and before you whip out the "no that's broken" argument, it's supposed to be fixed in perl 6.
Please do a little more research before you post all this FUD, people have thought of these "horrible problems" before.Re:reusable toolchain?
Alias on 2008-01-29T21:47:57
> In other cases, if it's safety you're after, in not running the compile time code, then theoretically you just use something like perl 5's Safe on all the macros and grammar extensions.
Only if you can solve the Halting Problem.
In Perl 5, even trivial Perl examples involve BEGIN blocks (use strict) and grammar modification (operator/operand switching).
This problem applies to Perl 5 to.
Simon Cozens has a never-released parser based on the Perl internal parser.
It works just fine, as long as the code compiles, and doesn't load modules that are broken or don't exist, and can compile in a Safe container, and... and... and...
Practically speaking, you end up in a situation of death by a thousand cuts.
There are so many little scenarios that aren't allowed that in practice you can't parse a useful subset of files at all.
Re:reusable toolchain?
chromatic on 2008-01-29T22:10:41
Only if you can solve the Halting Problem.No version of the Perl compiler or processor for any version of the language attempts to solve the Halting Problem. They tend to do a pretty good job on most reasonably correct code (and plenty of unreasonably incorrect) code as well. You don't need to solve the Halting Problem. You only need to decide if it's worth it at any particular point to Halt and say "Sorry, I'm not going to continue processing from here," and you can do that if you control the runloop.
Now if it's possible to modify Perl 6's grammar lexically within Perl 6 programs by writing only Perl 6 code, then it follows that Perl 6 must be able to parse Perl 6 grammars and have a pretty good idea of what they do. The rules are fairly declarative, after all.
I agree that it's likely possible for someone curious, malicious, or incompetent to write a grammar in such a way that you can't predict the effects of things (storing grammar rules in an
evalrun conditionally thanks torandfor example), but I'm not sure anyone thinks thatBEGIN { POSIX::_exit() if rand() < 0.5 };is a smart thing to put in Perl 5 code either.Re:reusable toolchain?
Alias on 2008-01-30T01:38:44
This pretty much covers my point.
If the code does anything remotely interesting or unusual, you have to abort parsing the document. (worse, you may have to do it after already having spent significant CPU trying).
Limiting yourself to documents that compile significantly reduces the types of tools you can use.
I guess in a way this entire post is something of a challenge to prove that a useful non-executing parser can be written for Perl 6.
Maybe I should formalize it at some point.
Re:reusable toolchain?
chromatic on 2008-01-30T01:53:06
If the code does anything remotely interesting or unusual...... and non-declarative, which I think you keep overlooking. While I agree that there are ways to write grammar actions that change parsing in unfortunate ways, grammars themselves look more or less statically decidable in ways that regular expressions aren't.
I won't suggest that they're quite as static as an EBNF grammar is, but they're much, much closer than the Perl 5 parser. It should be possible to identify arity and precedence without actually executing code, which clears up a couple of the thornier problems in parsing Perl 5.
Re:reusable toolchain?
Alias on 2008-01-30T02:52:41
Well, I'm assuming that "interesting an unusual" things will be non-declarative.
Anything declarative becomes "normal" for Perl 6.
Grammar changes have two overlapping issues.
There's the BEGIN problem. Lets assume that isn't a problem because grammar changes are declarative and decidable.
The secondary problem for grammar changes is how to (and if you can) handle syntactic and semantic modelling for the resulting document in such a way as to allow for stuff like $document->find('comments');
Re:reusable toolchain?
nothingmuch on 2008-01-29T23:20:43
The halting problem only applies to code you cannot introspect.
If you have a function, and that's all, then you can't find out what's in it.
But given a compiled optree, you have much more information.
If you parse the BEGIN { } block under the current rules, then you wind up with an optree which you can then examine, to see what it does.
As for simon's project - perl 5's parser was never designed to make this easy, it was designed to emit an interpreter optimized optree. This is very different from the design goals of perl 6's toolchain.Re:reusable toolchain?
Alias on 2008-01-30T00:40:31
We probably need to escalate this to a formally trained mathematician here, but as I understand it, it applies to any case with "arbitrary" code whether introspectable or not.
You CAN prove something will finish in finite time, you just can't prove how long that finite time is, which may be longer than the heat death of the universe.
Re:reusable toolchain?
Aristotle on 2008-01-31T09:51:48
Errm, you would have done well to follow your own advice and stop at the end of the first sentence.
:-) Re:reusable toolchain?
nothingmuch on 2008-02-02T17:17:35
No, that is just plain wrong.
You can very easily prove that the program 1 + 1 will return in finite, and short time, and it has nothing to do with heat death of the universe (very long != infinity).
Re:One True Grammar
Abigail on 2008-01-29T16:17:42
The mutable Perl 6 grammar scares me too, but I'm hoping that people never use it and it doesn't become a problem.Yet, whenever I raise my biggest objection I have against Perl6 (meaningful whitespace), I always get thrown back "well, you can change the grammar you know...".
Re:One True Grammar
schwern on 2008-01-29T18:01:00
Yeah, I was just going to say that. "Just change the grammar" was used to end Perl 6 language debate sort of like "God works in mysterious ways" is used to end religious debate.
Thankfully I haven't seen it come up as much lately. Maybe folks are starting to realize that easily mutable grammars are a powerful and awesome tool but not the sort of thing you want every kid on the block to use.Re:One True Grammar
Alias on 2008-01-29T21:49:27
I see multiple implementations as a strength.
At some point you simply HAVE to be able to have an IronPerl6 and JPerl6 simply for long term language flexibility and health.Re:One True Grammar
brian_d_foy on 2008-01-29T22:11:27
I thought that was the point of parrot---you didn't need different implementations if you had the byte code.
I don't see different implementations as necessary to anything. Some people might like it, but in reality people will code to the implementation's features. It happens in Java, Javascript, Lisp, Smalltalk, and probably a lot of others that I haven't used. The conversations at the pub are about who supports what and what you have to do to make good code on one implementation work on another.
It's not something I'm looking forward too.
Re:One True Grammar
Alias on 2008-01-30T02:56:31
There's going to be many cases where people are already standardised on a particular environment, typically CLI/.Net or JVM.
They already have management infrastructure in place to deal with it, and if you can fit in with the existing it's a huge plus.
Not to mention that if you can run on multiple implementations it's a huge advantage ecologically speaking, you can respond to environmental disasters much better.
(Just look at how Catalyst dealt with the Class::DBI disaster compared to Maypole)Re:One True Grammar
brian_d_foy on 2008-01-30T03:22:37
Catalyst dealt with the Class::DBI disaster by using something different. That's not how I want people to deal with Perl.:) Re:One True Grammar
Alias on 2008-01-30T04:21:25
Similar analogy...
How is some users of Python/Ruby dealing with their C implementations can be slow?
By switching to JRuby/IronPython/etc...
Similarly Catalyst survived by leaving Class::DBI, but nobody that was using Catalyst had to leave Catalyst. The same cannot be said for Maypole.
Re:Separate grammar from BEGIN blocks?
nothingmuch on 2008-01-29T19:06:55
The problem is that the grammar is code itself.Re:Separate grammar from BEGIN blocks?
Alias on 2008-01-29T21:51:05
That may not be a problem if the code is never run...
If "=" is mapped to sub equals, you shouldn't need to run equals while parsing, right?
Re:Separate grammar from BEGIN blocks?
nothingmuch on 2008-01-29T23:25:14
You don't understand. The grammar itself is a class. The parsing primitives are methods. It is actual Perl 6 code that does the parsing, you don't get a BNF for it.Re:Separate grammar from BEGIN blocks?
Alias on 2008-01-30T03:12:05
The parsing primitives are controllable and, fortunately, NOT arbitrary code.
And so they don't present a decidability problem, at least for the parsing primitives that are available out the box.
If we DO allow arbitrary grammar modifications, and those modifications will result in the execution of arbitrary code, well then we're back into the decidability quagmire again.
In fact, if arbitrary grammar chances are trivially associational with arbitrary compile-time execution then that should pretty much clearly prove that non-final grammars make the language undecidable for Perl 6, and we don't even need to introduce the problem of how to modelling arbitrary syntax and semantics.
Re:Separate grammar from BEGIN blocks?
nothingmuch on 2008-01-30T12:12:05
Go over the synopses again then... You can definitely define grammars in terms of code.
However, as long as this is trusted code, which you can determine by introspecting it, I don't think this is a problem.
Re:Really? It scares you that badly? Wow.
Alias on 2008-01-29T22:06:44
> I see no reason why there can't be a standard PPI6 - it is just the standard grammar. Done.
There is a VAST gap between anyone thinks is true and what they can prove.
Personally I DO see reasons why, plenty. Because I spent three years wrestling with them in Perl 5's grammar, several of which are based on mathematically provable impossibilities. And these grammar problems remain in Perl 6 unchanged.
You can't just invoke the "standard grammar" as some kind of magic cure-all.
SOMEONE has to eventually write this stuff, and at the moment the default expectation of many people seems to be on me.
> I think it would be great to be able to have parsers/editors/refactor-ers that can "statically" (whatever that means)
"Static" essentially means "context-free". That is, you can treat an arbitrary Perl source code file as a document, and parse it without needing to also parse any other documents.
> When time permits, or the problem requires it, or when I get bored I may try to modify the grammar to fit my need.
And this is EXACTLY what worries me.
If it is EAST to modify the grammar or worse becomes fashionable to do so, then we end up in a world of mess because an even smaller percentage of existing documents can be parsed.
> And if the tools aren't capable of being extended, well, then I'd seek out new tools.
This assumes that 1) Other tools exist 2) Such tools are possible to write at all.
> Here's where I put words in your mouth... you want "easy to parse," "standardized," "refactorable," syntax and you are scared that somebody will extend the grammar, so we shouldn't be able to extend the grammar.
No. I don't want it to be impossible to extend the grammar.
I see a lot of (theoreticaly at least) value in things like "use physics".
What I do see is a giant shiny thing that people will INEVITABLY want to use despite the fact it will make your code impossible to work with in anything more complex than Notepad.
So I want it to be HARD to modify the grammar.
Or at least I want something that triggers flashing lights and air raid sirens that you need to toggle before you are allowed to modify the grammar.
Re:Really? It scares you that badly? Wow.
rhandom on 2008-01-30T00:22:08
I am fully aware of your work with PPI on Perl 5. I am aware of the "proof" that it is impossible to statically parse Perl 5. I think they are amazing accomplishments. That said, I haven't had any need for PPI on Perl 5 and I think that it is obvious that BEGIN blocks potentially make it impossible to parse Perl 5. I believe your zero-ary sub followed by regex issue is actually taken care of in Perl 6 - at least the standard parser will be able to handle it.Sure I can. If I write standard perl 6, then I can use the standard grammar and get a parsed tree back... Like magic... Just like I said in my first post. If I don't write standard perl 6 then all bets are off, just like I said in my first post.You can't just invoke the "standard grammar" as some kind of magic cure-all.Um, no. The standard grammar parses standard perl 6. I don't really expect anything of you. I don't intend this as a slight. I just really don't have any requirements placed on you....at the moment the default expectation of many people seems to be on me. If you want to refactor your code or have method autocompletion, or check the code for security problems it is an impossible dream to analyze the code as an isolated document. You can't do it. You couldn't do it in Perl 5. You can't do it in Perl 6. If you allow inclusion of other documents, more things become possible.you can treat an arbitrary Perl source code file as a document, and parse it without needing to also parse any other documents.
If all you want is to syntax highlight, then you don't need a parsed document at all. You are actually better off without having to parse the document as it is nice to have syntax highlighting while you are creating an as of yet uncompilable document. The last thing I want is for the syntax highlighting to disappear and reappear while I type because the syntax goes in and out of being correct and incorrect.This was my fault for stating or implying that I need these tools. I don't. My emacs highlights perl 6 documents just fine as it is RIGHT NOW. It even completes methods for me too, albeit in a non-class-associative way. I'm productive enough right now in Perl 5 and I'll be productive enough in Perl 6. I'm not sure why not having a parsed document forces us down to the level of notepad. Of course having PPI style tools could possibly help me be more productive - but there is no guarantee that they would even if the grammar was static.> And if the tools aren't capable of being extended, well, then I'd seek out new tools. This assumes that 1) Other tools exist 2) Such tools are possible to write at all.To summarize with a run-on sentence: You want it to be difficult for an unquantifiable future developers, to potentially modify an un-finalized grammar, for an un-finished language, because the developers might possibly extend the grammar in a way that might possibly be harder to parse than the as yet-un-finished standard grammar, that might not be possible to statically parse, by a tool that doesn't yet exist, that hasn't been requested or at very least isn't required by the community and may or may not have any real utility at all, because writing the tool was barely possible for Perl 5, and writing the tool would be hard if not impossible for you to do.If it is EAST to modify the grammar or worse becomes fashionable to do so, then we end up in a world of mess because an even smaller percentage of existing documents can be parsed...
...So I want it to be HARD to modify the grammar...
To sum up my summarization: You want to cripple Perl 6 because it might be hard for your non-existing tool to parse it.
I think it should be easy to change the grammar. I'm glad Larry does too. I think amazing things will come of the ability to do so. I think it is premature to think that hobbling Perl 6 will provide more benefit than allowing for new design ideas. Nobody knew what would happen with CPAN when it was introduced. Modifyable grammars may fall on their face but I wouldn't bet on it.Re:Really? It scares you that badly? Wow.
Alias on 2008-01-30T01:00:39
> Like magic...
There is no magic, just complexity we don't understand enough.
> I just really don't have any requirements placed on you.
I didn't say YOU, I said "many people". I seem to get those sorts of comments about once a week or so.
And from about 50% of everyone that has used perlcritic or something else PPI-based.
> If you want to refactor your code or have method autocompletion, or check the code for security problems it is an impossible dream to analyze the code as an isolated document.
Pretty much all languages with simplistic grammars (Java/Python) can do this with relative ease.
Also, you don't have to do it perfectly. Approximations are able to be approximately as useful as a complete model.
> To summarize with a run-on sentence...
This comes across as dismissive and laden with uncertainties, but you pretty much nailed it.
It's reasonable to predict they will exist and be mostly like developers of other languages. Humans, with hobbies and jobs and preferences similar to other developers.
The grammar is unfinalized, but largely complete.
The language is incomplete, but hasn't stopped is discussing it for the last several years.
There's no "might" about it being not statically parsable.
The tool already exists according to you (the Perl 6 parser).
The tools have been CONTINUOUSLY requested, certainly to me, and clearly consider it useful. Personally, I've certain found my version synchronization tool and my SQL injection attack detector useful in real world scenarios.
The number of people requesting it demonstrated a need.
The fact it was barely possibly in Perl 5 is the REASON we should be treating this properly in Perl 6.
To counter-summarize: I want to cripple Perl 6 so that it is possible for ANYONE to write tools for it.
You might be brilliant enough to productively work on 100,000 line projects written by other people with nothing more than emacs, but I'm not.
Having effective developer tools is essential for being productive in a language across the full set of all language developers. If they weren't needed, then we wouldn't see the correlation that the best developers are generally the ones WRITING the tools to automate themselves out of development tasks.Re:Really? It scares you that badly? Wow.
rhandom on 2008-01-30T05:35:43
All of my jobs to date have met this criteria. Most of the jobs have been "pure perl" shops with 3 to 20 perl developers. We are hiring now in Orem Utah if anybody is interested. And emacs and vim are still the best tools for the job where I work. Not to abuse perl critic - but it helps little in what we are doing, whereas test suites and Devel::Cover help considerably. The Perl 6 seems more likely to make Devel::Cover possible than Perl Critic.You might be brilliant enough to productively work on 100,000 line projects written by other people with nothing more than emacs...
Re:Smart tools
Alias on 2008-01-29T22:16:23
You've outlined the situation perfectly.
Perl 6 breaks a ton of existing tools, while relying on the existance of new tools which everybody assumes will exist but nobody has actually proven can be written.
Re:Smart tools
rhandom on 2008-01-30T00:31:09
Besides the fact that the existing tools were written before Perl 6, there is no guarantee they would work with Perl 6 if grammar modifications were disabled. There is no existing tool today that works with perl 6 (other than basic syntax highlighting of various editors). Your entire basis in this thread is about the creation of "new tools which everybody assumes will exist but nobody has actually proven can be written.Perl 6 breaks a ton of existing tools,Re:Smart tools
Alias on 2008-01-30T02:59:29
I'm talking about the methodologies used to implement the tools.
If there is an equivalent environment, then it's reasonable to predict that an equivalent methodology can be used to achieve an equivalent result.
The code then just becomes an implementation detail.
I don't understand why you are attacking my in a discussion about a future language for talking future tools for that future language.Re:Smart tools
rhandom on 2008-01-30T05:27:55
I apologize if you felt a personal attack. Some of my comments may have been too heavy. But you surely expected some heavy responses when you attacked the future language. Still I apologize.I don't understand why you are attacking me in a discussion about a future language for talking about future tools for that future language.
Lets assume that we don't want to hobble perl6 before people have had a chance to evolve it -- yet we still want tools that can safely analyze non-trivial applications of it, without running those applications -- not even the "BEGIN" blocks. What would be required?
The answer is probably that you'd need to require any grammar modification to be accompanied by some meta data. Perhaps one could say that grammar modification is only permitted by a "used" module, and that it is allowed to modify the grammer only of its user, not itself. This would mean that the code analyser would need to see the meta data associated with the "use Foo;" statement, but not necessarily the code associated with module Foo.
This proposal might hobble perl-6 too much. So there's a second fallback: define a dielect (subset) of perl that enforces this rule. If you're writing perl6 code yourself, and so know that you want to use the tool, then you'd only permit yourself to use modules that conform the the appropriate meta-data spec (C6PAN modules might be tagged that they support it). A lint tool that sees you using a module that doesn't contain the meta data (minimal: "this module doesn't mess with it's user's grammar" tag) would issue an error.
Summary: I agree that unconstrained perl6 will be so dynamic that it is impossible to analyze statically, but my gut tells me that it will be possible to define a tame subset that can be analyzed. Library writers, and JAPHs, and golfers, would use wild-perl; corporate enterprise users would use tame-perl.
Re:Taming the Beast
Alias on 2008-01-30T03:05:35
I concur completely with both your conclusions and suggested adaptations.
This is pretty much what I'm advocating.
The key differentiation is perhaps that I strong think we need to define and support this subset/strict form of Perl 6 from day one and have it a considered part of the language, rather than try to hack something afterwards.
To quote Larry, "Why do people seem to keep thinking I'm only creating one language here".
From these N languages he is allowing, we need to at some point nail down an identifiable "Standard Perl 6" we can actually target development and analysis tools at.
If we know this exists early, then we can make sure all the lowest-level platform-abstraction and toolchain modules are all compliant with this subset, so that it won't potentially compromise downstream dependencies.
Re:Taming the Beast
dpuu on 2008-01-30T06:24:14
I guess the place where I disagree is the "have it considered part of the language" bit (other than that, by definition, a subset is part of the whole).My background is ASICs. The history of semiconductor design is one of increasing abstraction while maintaining a path to implementation (i.e. to manufacturable hardware). Many years ago, Verilog and VHDL appeared as hardware modeling languages. Tool vendors saw opportunity for profit if they could convert "models" to "implementation", and they started selling tools that could synthesize subsets of the languages down to gates. The "Synthesizable Subset" became a de-facto standard: it was the subset that you used if you wanted you code to synthesize across the tools from multiple vendors. More recently, SystemVerilog and SystemC are following a similar path.
Not everything can be defined using the synthesizable subset. Analog components, and complex cores, would be defined as "black boxes", with behavioral models (i.e. full language models -- or even models written in other languages such as C++) for simulation; plus meta data to define the relevant properties to the tools.
I would see the same path being followed for Perl6. Language designers and library implementors will use the whole language. People, such as yourself, will create tools that understand some useful subset. If other people agree that the subset is useful then they will limit themselves to that tool; and module authors will have an incentive (users!) to create the meta data. The exact form of meta data need not be part of Perl. It could be a separate domain-specific language -- though one could argue that Perl6 should have the hooks to allow it to be associated with the code (perhaps via POD).
So I say, don't wait for the subset to be blessed by @larry before creating the tool. Pick some arbitrary subset (e.g. pugs) and tell people of all the cool things they can do if they restrict themselves to just those features. If it's not cool enough, then add more features! I'd hope that a symbiotic relationship could form between programmer-tools and language-design.
Re:Taming the Beast
Alias on 2008-01-30T09:58:17
Thanks for the great precedent.
The thing in this case is that the language still hasn't been finalized, and if at all possible I'd like this subset blessed and supported internally.
For example, not just detailing the subset but actually having the parser finalize the grammar class by default, or something, so that the subset is both provable and enforcable.
And if that fails, well THEN we look at the alternatives...Re:Taming the Beast
dpuu on 2008-01-30T17:35:27
I'd be tempted to go down the road of being explicit -- avoids the political pain of fighting for the subset. In perl5, people are used to saying "use strict" at the top of every file. You could create a "use strict::static" module that (a) tells your tool that the user's module should be analyzable; and (b) modifies the grammar to eliminate the ability to modify the grammar. If the core language doesn't allow you to write such a module then you have a strong case to argue to tweak the language; but otherwise you can exist somewhat independently.That said, a good way to promote it would be to have the (an) implementation of perl6 were written in this safe subset. A statically analyzable language is easier to optimize at compile-time, so there's an objective technical reason for doing so.
Re:Taming the Beast
phaylon on 2008-01-30T19:08:17
Do you mean something like the Scheme standard revisioning process for a "Standard Perl"? That could actually be nice. Especially since we have a (assumably (is that even a word?)) larger userbase and a (also assumably) more practical than academic POV.
Re:Taming the Beast
Aristotle on 2008-01-31T09:50:24
(The word you were looking for is “presumably.”)
Re:Chris Dolan's unintentional response from Jan 2
tron on 2008-01-30T08:44:51
I think that pretty much nails it down.
In my opinion: If Perl6 does not make it easy to create tools like Eclipse or (though I don't like MS) Visual C++ for it and have by default - and without any extra effort by the programmer like adding meta data - the vast majority of available code ready for use in them, then it won't be successful with the masses.
It's not like ten or fifteen years ago when emacs ruled - today people are used to use IDEs offering advanced code completion, refactoring support and the like. If they don't get that for a language they simply won't consider using the language.
From what I've seen (and based on what I understand from what I've seen;-) Perl6 will be a great language - but it'll only live in the shadows without the right tools... Re:Chris Dolan's unintentional response from Jan 2
rhandom on 2008-01-30T16:13:10
This nails down my opinions about this article as well. My opinions are against people who are not @Larry or who haven't contributed more than a few patches (if even that), who speak as if in behalf of the community saying what should be done or predicting the sad future if their demands aren't met. I apologize to you because this rant is in response to your article when it really applies to a much larger set of postings across the years. Of course people are allowed their opinions - but I think the volunteers that are putting Perl together sort of deserve to follow their course unhindered.If Perl6...Who is Perl 6 being built for. It is being built for the people who love it by the people who love it. There are no companies or foundations dumping large amounts of money into it. Perl the language is a labor or love. And whether we create tools or not - Perl will always be in the shadows. That doesn't mean I'll love it less. The industry has shown time and time again, that just because someone builds something better (for an arbitrary person's definition of better), that doesn't guarantee widespread adoption.but it'll only live in the shadows without the right tools...
Perl 6 is and will be great. But there is no reason to believe that because it is great, that people will drop whatever they are using and flock to it. That won't happen. The question is, does that change your valuation of Perl.
You are welcome to build and use Perl because you think people will flock to it. I use Perl 5 because it helps me to get my job done, scales well code-base wise, scales well server side, and is generally a joy to work with. Everything I've seen about Perl 6 affirms that this trend will continue.
Thank your for your response, because it is one more note reminding me that I need to get in and contribute code to Perl 6.Re:Chris Dolan's unintentional response from Jan 2
tron on 2008-01-30T17:34:47
Who is Perl 6 being built for. It is being built for the people who love it by the people who love it.Yep. But I'm pretty sure more people would love it if some tools would ease their burden of tedious day-to-day work like code refactoring or make it easier just to take the first steps in a new programming language.
As someone else replied to you: You might be brilliant enough to productively work on 100,000 line projects written by other people with nothing more than emacs, but I'm not. That last remark certainly applies to me as well, no matter how much I like my vim.
;-) And whether we create tools or not - Perl will always be in the shadows.I don't actually see why I should take that for granted. Tools, of course, do not guarantee success - but they certainly help.
Maybe you don't like those fastidious milksops right from university who didn't learn their craft just with ed and who can only program if their IDE tells them what to do (well, there's some kind of exaggeration in that...) and just don't want to see them hanging around in the perl community? But there are many of them! And maybe I'm more of an evangelist kind of person than you - but I'd love to see a more widespread use of Perl6 if only it increases my chances for working on Perl projects with others.Thank your for your response, because it is one more note reminding me that I need to get in and contribute code to Perl 6.Well, even if that's the only effect my post had it was worthwhile
;-) Re:Chris Dolan's unintentional response from Jan 2
chromatic on 2008-01-30T19:16:06
Of course people are allowed their opinions - but I think the volunteers that are putting Perl together sort of deserve to follow their course unhindered.I don't know about the word "unhindered", but I will say this.
I thought about a problem in OO code for a couple of years and argued about it with friends and family and colleagues for a couple of years and, when Allison handed me the Smalltalk Traits paper and said "This is really close to what I think you want", I was finally able to convince Larry and Damian that existing OO systems had a real problem and that what would become roles was a real solution.
I don't know if the other designers would have found a similar solution in time. I do know that it took a lot of work to get a feature I wanted in the language, but we're all better off for it. That's the way many features get in community-driven projects: the people who really want them enough to work for them make them.
Re:Chris Dolan's unintentional response from Jan 2
rhandom on 2008-01-30T19:53:36
Yes - unhindered is too strong. The feeling I was trying to get to was "please stop complaining or changing perl and get in and do something."
Code speaks louder than words, and when you or Allison or Patrick or Jerry or any of the other current major contributors to Perl 6 find design deficiencies, you should have more clout and say in changing the language that those (like me) who up to date have stood on the sidelines.
Even though I've just been an observer, I've still grown tired of the rhetoric in and out of the community saying "Perl 6 will be bad because/unless..." For those in the community I know it is just because they want to make sure that Perl 6 really is great -- but posting inflammatory (whether intentioned or not) instead of going through the mailing lists or irc channels and asking Larry directly serves little to help the community.
I know Adam intended well. The method of delivery just seemed a poor choice. Especially in light of Moritz asking the exact same thing and getting a nice explanation from Larry.
Oh - and by the way. Thanks chromatic.Re:Chris Dolan's unintentional response from Jan 2
chromatic on 2008-01-30T20:49:09
... you should have more clout and say in changing the language that those (like me) who up to date have stood on the sidelines. I don't think that's necessarily true. While there is a distinction between committer and contributor, if you showed up with a test or documentation or code patch that was obviously right, I'd apply it like one of my own patches.
There is a big difference between complaining out of band and contributing, but I have reminded myself in the past week that:
- Some people just like to complain.
- Some people enjoy complaining more than feeling scared or concerned.
- Some people just like to make trouble.
I can do very little to help any of those people, so it's better for me to spend my time doing productive things and encouraging everyone else to participate in productive ways, such as on the mailing lists.
Re:Chris Dolan's unintentional response from Jan 2
Alias on 2008-01-31T03:11:17
> Who is Perl 6 being built for. It is being built for the people who love it by the people who love it. There are no companies or foundations dumping large amounts of money into it.
There is very little point in building an awesome language that we like using if nobody can be paid to do it.
If you can't creating a suitably strong economic proposition for the use of a language (and companies are, after all, almost purely economic beasts) and nobody can get a job working with the language, then the language is either a failure or merely a toy.
Ultimately, it comes down to productivity. Language design shouldn't merely be aa beauty contest. It's about making the most productive and efficient way of translating a developer's intent into machine-executable code.
Re:Chris Dolan's unintentional response from Jan 2
chromatic on 2008-01-31T04:41:57
There is very little point in building an awesome language that we like using if nobody can be paid to do it.By that metric, Perl 5's a failure because TPF can't seem to convince companies outside of the usual suspects (including Stonehenge, O'Reilly, ActiveState, MSDW, Ticketmaster, and the Beeb) to chip in cash to fund its development.
The traditional economics where businesses shell out modest piles of money for things that help them make bigs piles of money don't seem to work out in favor of the volunteers who create the things that help big companies make big piles of money. I've come to a sort of piece about that, where every few months I go off on a tear to call those companies antisocial, selfish leeches. The rest of the time I console myself with telling them their opinions about the software I write don't matter unless they somehow find a way to contribute to the writing of that software.
That plan seems to work out.
Re:Chris Dolan's unintentional response from Jan 2
grinder on 2008-01-31T14:14:14
(I think the word you were looking for was "peace":) Re:Chris Dolan's unintentional response from Jan 2
chromatic on 2008-01-31T18:00:58
True. At that hour, the word I should have been looking for was "Goodnight!"
Re:Chris Dolan's unintentional response from Jan 2
Alias on 2008-01-31T23:19:09
> By that metric, Perl 5's a failure because TPF can't seem to convince companies outside of the usual suspects (including Stonehenge, O'Reilly, ActiveState, MSDW, Ticketmaster, and the Beeb) to chip in cash to fund its development.
I wasn't referring to being paid to work ON it, I was referring to being paid to work WITH it.
Funding language development is a WHOLE different ball game...Re:Chris Dolan's unintentional response from Jan 2
nothingmuch on 2008-02-01T20:31:36
You are still working using the inherently flawed assumption that the best tools are written by reparsing Java/Perl 5/Perl 6 yourself, instead of reusing existing tools.
While that might be easy (perchance, easier, since the language is so simple) in the case of Java, or our only choice with Perl 5 due to practical limitations, if you have access to real meta data about the source code at all levels (text, parse tree, ast, runtime optree, emitter tree, etc) you are getting a lot more value.
There is lots of history besides Java, much more worth looking into. Smalltalk (especially Squeak), LISP/Scheme, forth etc are all good examples. Despite the fact that these languages have a very simple syntax (much simpler than even Java), they have established a culture where nobody bothers with the syntax anymore, even though it's so simple to do things with the right structures instead.
Re:Chris Dolan's unintentional response from Jan 2
chromatic on 2008-02-02T20:43:11
You are still working using the inherently flawed assumption that the best tools are written by reparsing Java/Perl 5/Perl 6 yourself, instead of reusing existing tools.... especially the language's own abstract trees. That technique gives Lisp and Scheme so much of their power and makes Smalltalk so malleable.
I'm not really that fit in the Perl 6 category, but I'd like to make some comment regarding the Scheme/Lisp way:
"I can write something in 4 days in Lisp that takes most people 20 in some other language. I just spend 3 days modifying Lisp to solve that type of problem, then 1 day solving the problem".
The anti-pattern here is that you end up going way beyond TMTOWTDI. You don't get two, or three, or five ways to do it. You end up with a different language for every single person and every single project.
Are you sure he and you meant the same thing? When I implement a different "sublanguage" in scheme, I don't change the syntax (eg. the reader). The typical parenthesis structure is (AIUI) _not_ the syntax, it's just the structure. The stuff we see, is only one way to tell the reader how to transform this into sexprs, which will then be executed.
For example, there are modules that do whitespace-indentation in lisp and scheme. You could just use a different reader to read that one file in that format. Nothing changed about the reader of the current, or anybody else's code.
Re:Some random notes
phaylon on 2008-01-30T19:50:54
As an addition: Most scheme systems (Can't remember R5RS at the moment) handle the syntactical differences (read: Macros or other procedures that don't work with the usual left-to-right evaluation) with syntax objects that can be imported. I wonder if Perl 6 could use something like "use syntax My::Company::Grammar".
Re:Some random notes
Alias on 2008-01-31T03:12:31
It's entirely possible I got the language wrong here... in retrospect that story may have been about SmallTalk perhaps?
Re:Some random notes
phaylon on 2008-02-06T16:49:33
I think the language could be the right one, without problem. It's just that developers of different languages have different semantics between the lines:)
Re:What Larry says...
Alias on 2008-01-31T03:14:00
I, of course, would still like to be placated by seeing someone actually do this...Re:What Larry says…
Aristotle on 2008-02-02T20:25:54
Note for anyone interested: the original post from Larry.
Note to rhandom: when you quote something that’s on the web, it is courteous to provide a link.
Re:What Larry says…
rhandom on 2008-02-02T21:57:00
Note to aristotle: at the time the post was made, the article wasn't yet posted on google groups, and I was unaware of www.nntp.perl.org which still most likely wouldn't have had the post yet as I posted relatively soon after Larry had replied to moritz. I did look for a link - but there wasn't one. If there had been I wouldn't have quoted so much of his reply.
However, you are very right. I should've later replied to my own node with a link. I think by that time I had resigned myself to the idea that if people wanted to see the full reply, they could look for a post from Larry on perl6-compiler around Jan 30 (details given in my post) in which ever way they normally access perl6-compiler.
