I recently received a bug report for HOP::Lexer, the lexer described in Higher Order Perl. One concern expressed in the bug report is that the lexer returns an array reference in the form [ TOKEN, VALUE ] with undef as the EOS (End Of Stream) value. However, if one doesn't use HOP::Parser but instead uses Parse::Yapp or similar tools, they expect a list in the form ( TOKEN, VALUE ) with the EOS indicated by ( '', undef ).
The suggested fix was to modify HOP::Lexer to optionally allow this. To my mind, this was the wrong fix. Since HOP is all about functional techniques -- applying functions to functions -- a much cleaner is to not create a special case, but to use a function!
In other words, what we have is this:
my $lexer = make_lexer( $iter, @input_tokens );
while ( my $next = $lexer->() ) {
my ( $token, $value ) = @$next;
...
}
But we can now do this:
my $lexer = make_lexer( $iter, @input_tokens );
my $yapp_lexer = sub {
my $token = $lexer->();
return ( '', undef ) unless $token;
return @$token;
};
use MyParser;
my $parser = MyParser->new;
my $value = $parser->YYParse( yylex => \&yapp_lexer, yyerror => \&error_sub );
That's the correct solution to this problem and, I think, an elegant one.
Of course, the other part of the bug report was line numbers from the lexer. I'll have to think about that.