[we could] not increment the reference count (and hope the SV is still around when the lvalue subroutine is called) -- Gerard Goossen
Hope and programming don't mix well usually. -- Rafaël Garcia-Suarez
The thread continued unabated this last fortnight, with a lot of discussion on how to improve INSTALL. The document attempts to address a number of conflicting requirements, which means that it won't be easy to get right.
http://xrl.us/tkb6
blead Steve Peters continued to analyse the reasons why Plagger failed
to build on blead. It turns out that some prerequisites also
fail on 5.8.8, so it's not as if it's completely blead's fault.
Stress testing stress http://xrl.us/tkb7
Term::ReadLine::Gnu TODO Andreas König found that Term::ReadLine::Gnu was whacked on
blead, which was due to what Nicholas Clark categorised as
an over-ambitious patch to copy SVs lazily in sv_setsv().
http://xrl.us/tar8
Ævar Arnfjörð Bjarmason was the first person to notice a problem
with the new _ (underscore) prototype (that indicates that the routine
operates on $_ by default).
If one defines a routine, such as:
sub odd (_) {
return $_ % 2
}
... then calling odd (as opposed to odd()) bypasses the pass-the-$_
mechanism. Rafaël Garcia-Suarez fixed this up straight away.
still in beta http://xrl.us/tkb8
s/// and hashes Rafaël was puzzled by a leak with substitutions and hashes and wondered if the problem was something to do with shared keys. Dave Mitchell realised that the problem was something else entirely, and reduced Rafaël's original problematic code down to a one-liner.
Dave's code showed that the problem was that as soon as one executed
a regular expression with an alternation (that is, /a|b/), a few
SVs were leaked each time. Yves Orton sighed, and admitted that it
was probably all his fault.
Dave pinpointed exactly where in the code they were leaking from. This was enough to help Yves, and he supplied a fix in short order. Which worked, and was thus applied.
And they lived happily unallocated ever after http://xrl.us/tkb9
Aaron Sherman took a look at the stock answer to this question (in perlfaq8) and dusted it down and made a few improvements. The change was promptly accepted by Rafaël.
For some strange reason, the thread then devolved into a long discussion about locales, Unicode and escaping characters that don't need to be escaped.
http://xrl.us/tkca
Pod::Coverage Steve Peters noticed that Pod::Coverage thought that lots of
routines weren't documented in blead. The routines in question
were all constant routines, and, thanks to Nicholas Clark's efforts
to reduce the amount of memory they use, they are now so small that
they escape Pod::Coverage's attention entirely. Nicholas suggested
one way of fixing this.
Jerry D. Hedden cooked up a torture test to provoke race conditions more easily with threads. The results exceeded his wildest dreams (or nightmares, take your pick).
Dave Mitchell gave Jerry some assistance, and after a while Jerry found the root of the problem that was responsible for nearly all the errors. Once this was fixed, only a couple remained, but Jerry had a feeling he knew why they were occurring.
http://xrl.us/tkcb
PL_perlio_fd_refcnt and PERL_TRACK_MEMPOOL Jan Dubois reported that things were still not playing nicely in PerlIO land when the right combination of configuration parameters was found. The problem was that the underlying memory allocation routine was receiving a pointer to a block of memory allocated by a higher-level allocation wrapper.
The higher level code is needed when threads come into play, where more detailed knowledge about which thread asked for the allocated memory becomes more important (especially on Win32). Jarkko Hietaniemi had an idea of how it could be fixed.
http://xrl.us/tkcc
Jerry hacked around a bit more, and found that a detached thread was
destructing itself after the main script had already ended, but didn't
know how to pin it down in more detail. After stepping through the
code he found a wonderful comment /* FIXME: treaddead ??? */ in
the routine at the top of the call stack.
Evidently he was onto something, and supplied a patch to fix it up, which Rafaël dutifully applied.
http://xrl.us/tkcd
Having pondered the metaphysical issues of FIXME comments in code for a while, Jerry grepped the source to see if there were any others lying in ambush... and discovered dozens.
He wondered whether it was worth going through the list and adding items to the TODO list as needed. Warnocked.
http://xrl.us/tkce
Recent code changes to blead caused linker failures on Win32. Both he and
Yves Orton independently discovered the fix. So Rafaël applied the change,
but Steve continued to note a crash in ext/XS/APItest/t/svsetsv.t.
Steve finally managed to get things straightened out with some particularly devious macro trickery.
pay no attention to the man behind the preprocessor http://xrl.us/tkcf
bignum breaks Getopt::Long It's not so much that one breaks the other, it's just that politely ignoring
each other can only get you so far. H.Merijn Brand uncovered a number of
problems when he needed 64 bit integers on a 32-bit CPU, so he just
added -Mbignum to the command-line, and was faintly surprised when
things fell apart.
There followed a long thread about implicit assumptions, containers
and content, the uselessness of ref($foo) eq 'REF'. In the middle,
Yves Orton proposed a patch to fix things up, and a nice overview of
why it's so difficult to write code that asks "can I use this reference
as an array?".
http://xrl.us/tkcg
Tels discovered that Benchmark fails under -Mbignum and filed
bug #40917. Sadahiro Tomoyuki determined what the problem was and
came up with a fix, which Steve Peters applied.
http://xrl.us/tkch
substr using mortals Gerard Goossen returned to a problem he raised in August:
http://xrl.us/tkci
with a patch that he proposed to fix the behaviour. Rafaël had quick look at Gerard's idea, but wondered if it wasn't just sweeping the underlying problem under the carpet until the end of the scope. Gerard explained that he saw there were two alternatives available and how to make them work.
Unfortunately, after a little thought, he realised that the first alternative couldn't work. Rafaël found another couple of problems with it as well, and hinted that the second alternative was possibly too fragile. It remains to be seen whether some sort of weak reference might do the trick.
http://xrl.us/tkcj
H.Merijn Brand did some house-keeping on Configure. First he refreshed the MC units.
http://xrl.us/tkck
and then went through to see which new variables that had been added to Configure in recent times had no sensible defaults for them. The platforms included Plan 9, Win32, VMS, Symbian and a few others.
He also wrote some code to ensure that changes in the future will come up on the radar much more quickly, rather than having to remember to do run some quickly-forgotten script, A number of porters pitched in and filled out most of the missing blanks.
http://xrl.us/tkcm
Class::MethodMaker breaks somewhere between 5.9.3 and 5.9.4 Steffen Schwigon noticed that this module, a pre-requisite for
several dozen modules on CPAN alone, fails on blead, from
5.9.4 onwards. Rafaël traced this to a change that outlawed
the lvalue use of GvNAME.
Steve Peters fixed things up, and started to work on getting ppport up to speed on the matter.
http://xrl.us/tkcn
ar command use a file to list the objects to add? Steve Hay was trying to teach ar to look at a file and use its
contents as a list of object names to add to a library, all for
the sake of reducing the length of a command line.
Nicholas Clark wondered if miniperl was usable at this point, in
which case it could be used to furnish basic xargs functionality.
http://xrl.us/tkco
John E. Malmberg was back in town, hacking perl on VMS. He was having trouble with the debugger, and he and Craig A. Berry thrashed out the issues.
http://xrl.us/tkcp
Later on he committed a patch, which provided some nifty functionality for running the debugger in a separate terminal windows. Craig A. Berry thought the patch was promising, although he had a couple of reservations about the implementation. Nevertheless, he applied the patch and promised to take the debugger out for a spin.
http://xrl.us/tkcq
DateTime::Format::Strptime Andreas Knig isolated the change that caused DateTime::Format::Strptime
to start falling over. Yves Orton realised that this was a symptom
of a bug in the regular expression engine, made the appropriate
fix, and added a test to make sure it stays fixed.
http://xrl.us/tkcr
The patch caused something else to fall over in the test suite. This turned out to be because Yves forgot to include all the relevant pieces needed.
http://xrl.us/tkcs
Andeas also caught the problem, leading Nicholas to comment that
t/harness needs to become fussier, since it misses things on
Win32 (where Yves develops) that only show up under TEST on
Unix.
http://xrl.us/tkct
Even though 5.005 threads have been completely excised from the
5.10 codebase, the Configure infrastructure still knows about
it, since it is used to generate the final Configure scripts
for both maint and blead. He did manage, however, to make
it shut up.
http://xrl.us/tkcu
Rafaël announced that he wanted to push 5.9.5 out the door, and asked the porters what they were working on, and what they thought would be ready soon.
Dave Mitchell admitted that he had broken a few things recently but was unlikely to find the time in the next few weeks to attempt the repairs.
Steve Peters wanted to make directory handles more useful, but was stuck on a problem with pipes hanging on OpenBSD, and was having trouble ruling Perl out before filing a bug with the OpenBSD kernel hackers.
H.Merijn Brand mentioned that there were a few Configure
issues he wanted to look into, and possibly an edit or two to
the README/INSTALL issues as raised by Michael G.
Schwern. H.Merijn's Configure issues reminded Andy
Dougherty about a problem with locales, GNU's make and Solaris,
but lacked a Solaris box to check it out.
Yves Orton mapped out his priorities, mainly bug fixes and making semi-broken things work sanely. One big item was that he wanted to get the regular expression engine to a state where custom engines could be dropped in, that would allow, for instance, test coverage analysis of regular expressions, or otherwise hacking on the engine without being forced to relink perl.
Steve Hay wants to get Visual C++ support finished, and
resolve the bug that prevents mod_perl 1.x from building
successfully. When Nicholas Clark mentioned the probable
cause of the problem, it reminded Rick Delaney about diddling
%INC to force a module to be reloaded (something that
mod_perl likes doing a lot). And that was enough for
Nicholas to track down the offending code, and fix it.
Jos I. Boumans provided the list of what modules need to go into core in order for CPANPLUS to be added in.
Philip M. Gollucci noted that there were still some 5.005 threads remnants dotted about in the code. Steve Hay thought that it was just harmless detritus, since the 5.005 threads implementation code had been removed over four years ago. Steve Hay thought that it should be cleaned up completely, if only to avoid further confusion in the future.
Andy Dougherty cautioned that further cleaning would only
add to the difficulty of merging blead back into maint.
H.Merijn agreed, saying that the current state of affairs
was pretty much optimal.
http://xrl.us/tkcv
Steve Hay discovered that this test reads its own source code, and saw that it managed to read only 4096 bytes of code and thus flunked the test. Steve thought the number was a little too coincidental to have been left to chance, and wondered where to start looking.
my lucky number http://xrl.us/tkcw
Steve Hay fixed up problems in posix.t with strftime hanging when VC8 was
used to build Perl. He traced it down to the fact that the C run-time
library didn't recognise %D as a valid format, and things went
downhill from there.
http://xrl.us/tkcx
He also tidied up the warnings about deprecated CRT function names in VC8,
and updated perltodo, because what he did was at best a stop-gap measure,
and someone needs to come up with something better.
http://xrl.us/tkcy
qr// benchmark Tels ran some benchmarks on compiled regexps and wondered what the numbers meant. A long discussion followed, developing on the concepts of implementing a "rule" feature for Perl 5, how compiled regexps behave, how to deal with combining precompiled regexps and best practices for building a big precompiled expression from a number of smaller fragments.
http://xrl.us/tkcz
A number of porters agreed that this would be useful, and in the end the necessary changes were committed to Win32's Makefile.
http://xrl.us/tkc2
Yves recent work to teach the regular expression optimiser about lookahead assertions threw some other assumptions about target lengths off. So he whipped up a patch to straighten things out. Bundled in with the patch were assorted improvements he'd made to the engine here and there.
http://xrl.us/tkc3
Now that the regular expression has a new recursion syntax (via
(?1)), Yves has been exercising them by converting old-style
(??{...} recursion patterns over to it, and in doing so discovered
a fundamental limitation, in that some times you want to recurse
back to a capture buffer that's relative to the current location,
not the beginning. So he extended (?1) to allow (?-1), and
made it work, too.
A new feature in every box http://xrl.us/tkc4
He also closed out bug #19049 and added relative backreferences, which
means you can say /foo (\w+) bar (\w+) \R1/ and the reference count
runs backwards from the current point, leftwards, to the nth capture
group.
I for one will find this handy http://xrl.us/tkc5
He also fixed up /a++b/ by teaching the optimiser to look for "ab"
before bringing the heavy artillery to bear.
http://xrl.us/tkc6
He also added regmust() to re.pm/re.xs, which exposes to the
client code what the optimiser considers to be the floating and
anchored strings of a potential match.
http://xrl.us/tkc7
In other news, Yves also reorganised the flags of the regexp
struct into two fields, one internal, and one external and applied
a consistent naming convention.
Separation of interest http://xrl.us/tkc8
local save/restore (#1716) Yves Orton went travelling back in time to have a look at some really olde
bugs. This earliest bug was an issue of magic. Yves thought that there should
be some way of getting things to work correctly for pos at least.
This reminded Dave Mitchell that he had been meaning to get around to
looking at how local and magic play together (or not), and told Yves
that it was probably on his personal TODO list. Ironically, the sig attached
to the reply was "Never do today what you can put off till tomorrow."
Magic, murder and the weather http://xrl.us/tkc9
Rafaël also retired another ancient bug: assigning undef to a signal
handler now restores the default handler.
http://xrl.us/tkda
Yves found another bug report concerning core dumping badness over a
regular expression that Steve Peters missed a while back when he closed
out several zillion bugs,and noted that it was fixed in blead.
http://xrl.us/tkdb
Yves explained that attempts to use variable width look-behind assertions (which aren't currently implemented in Perl, although Yves will no doubt implement them next week) cause the HERE mark to point at the wrong place.
This is because by the time the parser has figured out that there is indeed a variable width look-behind assertion, it no lacks the neurons to remember where it actually occurred. Yves suggested that the best way forward would be to drop the HERE marker in this context.
He should know http://xrl.us/tkdc
Yves classified this bug a more of a parsing issue than a regular
expression issue (the fact that ${10} doesn't behave as it should
(like $10)). He added a TODO test for someone else to address.
http://xrl.us/tkdd
\G with /h results in infinite loop in 5.6 and later (#6893) Yves thought that this bug stank, since it involved people playing
around with things that the regular expression engine would rather
keep to itself. He fixed the problem, but this caused a test in
Text::Balanced to fail. But then again, as the man page of that
module says, "There are undoubtedly serious bugs lurking somewhere
in this code".
http://xrl.us/tkde
With another round of patching, he managed to get thing into good shape.
http://xrl.us/tkdf
s/\s$word\s/ /gi and $& (#18209) Yves could not reproduce this problem, either in maint or blead.
Just to be on the safe side, he wrote a test case to make sure someone finds
out about it if it ever comes back again.
http://xrl.us/tkdg
(a|b)* cannot match a string longer than 2**15-1 characters (#18268) Yves confirmed that this was indeed a hard limit of the regular expression engine and couldn't see how it could be fixed, so he added a couple of tests to the test suite, and marked them TODO.
http://xrl.us/tkdh
/(.*)[bc]/ 10000 times slower in 5.8.0 vs 5.6.1 (#22395) This was caused by an over-pessimisation in change #14115. Yves optimised the common case to make it go 10000 time faster again.
http://xrl.us/tkdi
He also made another regular expression go speedy fast again (bug in regexp m/(.*?)b/ig) (#25822)
http://xrl.us/tkdj
Encode::is_utf8 on tainted UTF8 string returns false (#32687) Rafaël fixed up the tainting logic of blead's copy of Encode and
suggested to Dan Kogai that he incorporate the change in his own version.
Paging Dan "Mr. Encode Maintainer" Kogai, white courtesy phone http://xrl.us/tkdk
$^R undefined on matches involving backreferences (#36909) Yves also fixed this bug. H.Merijn yearned to clean up the white space as well, thought about what Dave Mitchell might say, and decided to do nothing.
http://xrl.us/tkdm
The above change also fixed a bug your humble summariser uncovered a couple of years ago. Now it's fixed. Yay Yves! ($^R value lost in (?:...)? constructs, bug #32840)
http://xrl.us/tkdn
Brian Candler posted a bug report in September concerning a regular expression
used to parse a particularly devious line in an Apache log file. Yves was
pretty sure that the problem was resolved in blead.
http://xrl.us/tkdo
Cwd::chdir() and handles (#38466) Ken Williams was astonished to learn that chdir may operate on file
handles. He updated the PathTools distribution.
"Holy crap, chdir() can accept a filehandle? I had no idea." http://xrl.us/tasi
Stas Bekman checked in to see what the status is with this bug.
Answer: fixed in blead, still fails in latest maint.
http://xrl.us/tkdp
=> (#40854) Dr. Ruud asked a fairly innocent question about bareword hash keys and the fat comma operator, and this generated a certain amount of traffic. John Peacock correctly identified the issue as being the fact that the leading zero of the bareword was triggering octal interpretation.
http://xrl.us/tkdq
find2perl -ls very slow on some LDAP-based getpwent systems (#40867) David Dyck found that find2perl's propensity to warm its caches
caused massive slowdowns on big LDAP directories, and wondered if
there would be any problems in adopting a lazy caching strategy.
Rafaël saw no problem.
do not do today... http://xrl.us/tkdr
pod2xxx requires Pod::* parsers to reset state (#40899) espie uncovered a problem with Pod::Man et al. using a single POD parser object to parse multiple files, which causes problems. Earlier files with POD errors influence the rendering of subsequent files. (This issue reared its head a few months back, when Russ Alberry was working on the podlators distribution).
http://xrl.us/tkds
Christopher J. Madsen discovered that reading fixed-length records
(for instance, with $/ = \10) and :encoding don't play well
together. Especially when converting between EBCDIC and UTF-8.
why am I not surprised? http://xrl.us/tkdt
Jerry D. Hedden found, much to Yves's dismay, that the Aho-Corasick patch was not thread-safe. When Yves looked at what needed to be done he threw up his hands in horror and wondered if there wasn't a better way of abstracted the differences between threaded an unthreaded builds.
The main problem is that the op-tree (which also contains regops) is shared among all threads, so detached threads tend to have more problems with heavy objects hanging off the op-tree. Dave Mitchell said that he had plans for a "copy-on-retrieve" technique for this problem, which would mean that a lot of these sorts of issues with threads would suddenly disappear.
http://xrl.us/tkdu
Similarly, Jerry also found that Perl_op_clear is not thread-safe (#40965).
http://xrl.us/tkdv
Date::Parse mysteriously lowercasing text when an unrelated variable is used. (#40957) Mumia W. encountered a curious bug involving sorts, closures and lc
which resulted in some very strange action at a distance.
and there was great puzzlement http://xrl.us/tkdw
-DNO_MATHOMS (#40963) Jerry D. Hedden found that you can't build perl without mathoms, despite what it says on the tin.
http://xrl.us/tkdx
Vaclav Ovsik built a perl with his bare hands on Win32, and noticed that he couldn't disable line-endings translations through the PerlIO layer.
http://xrl.us/tkdy
pos() in regexp (#40989) Christian Mittag had the decency to produce one of the tiniest programs possible to provoke a core dump, with the following code:
$_="a\x{111}\x{111}a\t";
s/\t/pos;' ';/eg;
Unfortunately no-one leapt to his rescue.
won't you come on down to my http://xrl.us/tkdz
For the first time in a quite some time, a significant reduction in the number of open bugs was observed. Yay! Congratulations to everyone who helped out.
5 open, 7 closed http://xrl.us/tkd2
7 open.... and *18* closed http://xrl.us/tkd3
We need a "choose random bug" link http://rt.perl.org/rt3/NoAuth/perl5/Overview.html
threads went to version 1.51 and on to version 1.52.
http://xrl.us/tkd4 http://xrl.us/tkd5
Module::Pluggable was offered to the core.
Jos, unplugged http://xrl.us/tkd6
Module::Load::Conditional made it to the core. Tels wondered about
core bloat, but I told him that there was nothing to worry about.
It's all good http://xrl.us/tkd7
IPC::Cmd also went past Go and collected $200.
http://xrl.us/tkd8
The "WHOA THERE" messages were toned down.
http://xrl.us/tkd9
Yves Orton was a busy man this week, in case you hadn't already realised. Some more of his handiwork included noticing that bug #6719 wasn't. It was
working as documented http://xrl.us/tkea
Bug #17542 is no longer confuses on blead,
http://xrl.us/tkeb
$` is now always correct after replacement (bug #19049)
http://xrl.us/tkec
and there are no more spurious undefs in @- and @+ during
(?{}) and (??{}) (bug #22614)
http://xrl.us/tked
In other news, Rafaël fixed up a problem with base.pm calling ->isa() (bug #24824)
http://xrl.us/tkee
and made tied variables work with .= <> in blead (bug #38631)
http://xrl.us/tkef
On the other hand, he didn't know what to do when $1 is bound
to an out-of-scope variable so he added an assert in the code, to
fail gracefully, (bug #38869)
http://xrl.us/tkeg
although he managed to make Tie::Memoize::EXISTS cache its value
(bug #39026),
http://xrl.us/tkeh
as well as ensuring that rcatline stringifies references (bug #39037)
(when it has to).
http://xrl.us/tkei
Adriano Ferreira closed out a Symbol::delete_package() bug (bug #34078) (a
documentation patch), since the documentation has already been patched,
http://xrl.us/tkej
and added some tests and documentation for &=, |= and ^= (bug #36689)
http://xrl.us/tkek
as well as an improvement to a documentation example for
File::Basename . (bug #40866)
http://xrl.us/tkem
Nicholas Clark managed to resolve the blead crash in
Perl_pp_entersub() (bug #40681) to his satisfaction by
tweaking the sv_setsv_flags macro's behaviour.
http://xrl.us/tken
Philippe found some inconsistencies in items formatted with
pod2html (bug #40851), but no pod2html specialists were
on hand to offer advice.
http://xrl.us/tkeo
Philip M. Gollucci thought that rt.perl.org needs a 5.9.5 category
for reporting bugs.
http://xrl.us/tkep
Mike Guy, man of infinite memory, recalled that list assignments with
lvalue undefs (such as ($foo, undef, $bar) = @list) was introduced
between 5.004_04 and 5.004_05.
http://xrl.us/tkeq
Buu wondered if you could create lexicals via other methods than my .
why? http://xrl.us/tker
Ken Williams wanted a regression test for the change to
ExtUtils::ParseXS for $filepathname .
Warnocked http://xrl.us/tkes
Ron Blaschke had some thoughts on Microsoft Visual C++ 8.0 (quite an essay, really), and Steve Hay commented that he had reached much the same conclusion.
http://xrl.us/tket
H.Merijn Brand and HP-UX 11.xx and GNU gcc can make the 5.8.x debugger dump core at will.
http://xrl.us/tkeu
Phil Pennock penned a message describing problems in IO::Socket::INET6 and
INET4 compatibility.
http://xrl.us/tkev
Andreas König noted that patch 28788 to bleadperl breaks
XML::Atom , so Steve Peters fixed things up.
http://xrl.us/tkew
POSIX::remove() deletes directories now, because the specification
says it should.
http://xrl.us/tkex
Text::Template fails with bleadperl and so it should, because it
uses \Q to quotemeta a string, and that string contains '\, and
quotemeta is now correctly doubling up the backslash.
http://xrl.us/tkey
Jarkko gave us a pointer to the Safer C Library.
they've done it now http://xrl.us/tkez
Nicholas wondered if it was worth warning about unused lexicals.
http://xrl.us/tke2
He also met up with an old familiar silly error, and as a result
Carp's cache should work better.
hand the man a brownie point http://xrl.us/tke3
Jarkko bestowed some portability updates upon perlhack,
http://xrl.us/tke4
and pointed the fire extinguisher at certain bits of code that the daily smokes were coughing over.
http://xrl.us/tke5
This summary was written by David Landgren. My apologies to those of you undergoing summary withdrawal symptoms. The reason there was no summary last week was that I spent all my tuits working on slides for the French Perl Workshop. If perl 5.10 isn't released before YAPC::Europe next year, I just might have to inflict my talk on another audience...
In the previous summary, I wrote that Brandon Black wrote Class::C3,
when in fact it was written by Stevan Little. Brandon is the current
maintainer. My apologies to both for the mix-up.
Weekly summaries are published on http://use.perl.org/ and posted on a mailing list, (subscription: perl5-summary-subscribe@perl.org ). The archive is at http://dev.perl.org/perl5/list-summaries/ . Corrections and comments are welcome.
If you found this summary useful, please consider contributing to the Perl Foundation to help support the development of Perl.