The biggest problem of PPI is that its slow. This causes any code depending on it such at Perl::Critic or the syntax highlighter of Padre to be slow. In the last couple of weeks Semuel Fomberg has been writing a replacement Tokenizer in C++.
See his status report.
I am sure he'll accept your help.
Through simple profiling, I got 30% in some simple hacking.
Re:How much faster?
gabor on 2009-03-01T09:07:42
I don't know how much faster the C++ implementation can be but currently it is reasonable to use PPI in Padre only for files that are less than 300 lines long.Re:How much faster?
simonflack on 2009-03-01T17:31:55
I recently wrote a custom XML syntax highlighter for Wx::STC. I found that if I tried to highlight more than a couple of hundred lines at a time it would be unacceptably slow. And even fewer lines if I waned to edit (and not just view) the document.
I got around this by only styling lines that were visible. I used GetEndStyled() to tell me where I should begin styling, instead of starting at pos=0 every time. I stored state in SetLineState() which coupled with GetStyleAt() gave me enough information to know whether I might need to backtrack a little when restyling. I would only style 200 chars for each EVT_STC_STYLENEEDED(). And I shortcut the highlighing if either (a) I went off the screen, or (b) my styling matched the existing style.
With all these optimisations I ended up with some nippy syntax highlighting. I haven't checked how Padre is using PPI for custom syntax highlighting. And for all I know, you are already using similar optimisations. But if you're not, I think you will need to because he STYLE_NEEDED event is raised on every key press, so you're going to want the event handler to return prety quickly. Even with a C++ tokeniser I don't think you'll be able to highlight entire documents quickly enough without impacting user experience.
Best of luck,
Simon
