A quiet week on perl5-porters, in which we learn that if we diddle with the environment, we eventually pay for the consequences, and Schwern saves the day by preventing the module testing infrastructure from falling apart.
David Dyck wrote in to say that Inline's make test was failing
in blead, due to a construct that applied the defined operator
to a hash. Rafael Garcia-Suarez summed up the problem succinctly:
$ perl5.8.7 -wle 'print defined %foo::'
$ bleadperl -wle 'print defined %foo::' 1
and opined that the use of defined(%hash) was discouraged anyway.
Dave Mitchell pointed out that for lexicals, a compile-time warning
is issued:
$ perl -wce 'my %foo; print defined %foo' defined(%hash) is deprecated at -e line 1. (Maybe you should just omit the defined()?)
Rafael traced the change in behaviour to patch #24660, in which
Nicholas Clark shaved four bytes of the size of an HV body on
32 bit platforms. And the patch comment even stated that one
side effect was to make defined %symbol_table:: always true.
Alas, it may be that this breaks Tk if the source code comments
are to be believed. So far, it doesn't appear that anyone has
investigated this possibliity.
http://xrl.us/hxs4
Curtis "Ovid" Poe worked out how to make Data::Dumper do something
clever like:
print Dumper($foo, @bar); # prints something like $foo = 'stuff'; @bar = ( 3, 17 );
(Data::Dumper::Simple already does this, albeit with a source filter).
This time, no source filter. The biggest problem is that this would
require bringing PadWalker into the core. And then...
Steve Peters related his difficulty in getting PadWalker to pass its
tests on Linux, which made him wonder what things might be like on more
exotic platforms. Robin Houston (the author of PadWalker) mentioned that
this has been fixed in version 0.13. Dave Mitchell noted how PadWalker,
relying as it does on undocumented perl internals, is very sensitive to
breakages between perl versions, is not robust, and should not be
included in core.
Rafael was more concerned about teaching Data::Dumper new tricks, a
justifiably ancient module which is probably better left alone.
Robin answered that the Perl debugger will make use of PadWalker if
it find it, and was concerned by the porters' view of its apparent
fragility. He admitted that this perception might be due to the fact
that he let it slide from August 2003 to August 2005, but that he is
now working hard to make it as robust and reliable as possible. He
didn't see any fundamental reason as to why robustness could not be
achieved.
Dave said that he wasn't criticising PadWalker per se, but that
the goal it is trying to achieve is fraught with peril -- source
filters, undocumented internal interfaces, lexicals, closures and other
subtleties -- such that even perl itself doesn't always get right.
Robin countered again, with a new version that resolved nearly all of
Dave's problems and then asked for a better name for the module,
perhaps something in the Devel or Lexical namespaces.
If you only have time to read one thread this week, this is a good 'un.
Ovid's journal entry discussing the Data::Dumper hack http://use.perl.org/user/Ovid/journal/26973
The thread on p5p http://xrl.us/hxs5
There are lots of warnings issued when one runs
C<make test.taintwarn>
(Gee, I didn't know that existed). Steven Schubiger noted that
many of them come from using $^X to determine the name of the perl
executable. Steven wondered what the best way to fix that would be
(such as a hardcoded ../perl) and also said, in fine understatement,
that some of the code seems rather difficult to untaint.
Steve Peters mentioned that searching for perl will be difficult
when your executable is named ponie. Rafael thought that $^X being
tainted was a good thing, and that it was not sufficient to untaint
it with a brute-force /(.*)/.
After taking a second look at the problem, Rafael came back and said
that he didn't really care about the warnings. The test.taintwarn
is there more to test the tainting mechanism itself, that is, tests
totally unrelated to tainting that begin to fail when tainting is
enabled. If it warns, but runs ok, that's enough.
The thread then went on a couple of tangents, from bringing
File::Which into core, to better deal with tainting, to the fact
that MacPerl was no longer supported.
The latter remark stung Chris Nandor into stating that a platform is not declared unsupported until such time as those developers who care about supporting Perl on that platform decide to call it a day. He stated that he did intend to release another version of MacPerl at some point in the future. 5.8.x came very close at one time, but then Real Life intervened. Nevertheless, another release is still quite possible.
John Malmberg, ever a glutton for punishment, said that he was trying to determine whether or not the taint code was being aggressive enough in tracking VMS-ish things susceptible to taintedness. Craig Berry said that a certain number of things were already being done, such as dropping image privileges (my VMS is too rusty to summarise what that means).
From a small patch, a mighty thread grows http://xrl.us/hxs6
Benjamin Franz was trying to install Test::Plan, and getting strange
failures:
%Config::Config is read-only
After looking at the code, he noticed simplified the problem down to the following, and posted it to the list:
use Config;
local *Config::STORE = sub { 1; };
(%Config is actually a tied hash). He went to the trouble of digging
through RT (the bug tracking system) and unearthed #35865 and #35865.
Michael Schwern replied that since Test::Plan is digging around in
Config's internals, this hardly constitutes a bug on the part of
Config, and invoked Tom Christiansen:
You are wicked and wrong to have broken inside and peeked at the
implementation and then relied upon it.
-- tchrist in <31832.969261130@chthon>
... and then showed how he solved the problem in ExtUtils::MakeMaker.
(Basically, one takes a copy of %Config). Benjamin forwarded the
information to the author of Test::Plan.
The lethal problem: http://xrl.us/hxs7
How EU:MM does it: http://search.cpan.org/src/MSCHWERN/ExtUtils-MakeMaker-6.30/lib/ExtUtils/MakeMaker/Config.pm
Steve Hay reported back to say that the protocol files on NT4 Workstation, 2000 Server, 2003 Server and XP Pro are all identical, save for copyright dates. So Steven Schubiger's test is in as change #25696. But then Rafael immediately had problems on Linux smokes and pulled it out. Steven went back to the drawing boards with a less ambitious test. Applied by Steve Peters.
It started in August http://xrl.us/hxs8
And continued in September http://xrl.us/hxs9
Which was summarised here http://dev.perl.org/perl5/list-summaries/2005/20050926.html
And then this week http://xrl.us/hxta
Robin Barker noted that Perl_io_close is used twice in the perl source.
In one location the return value is used, in the second, it is ignored.
So he proposed fixing embed.fnc to fix the warning when it is not
used. Rafael noted that it was invoked during SV destruction, and thought
it might be nice to do something if the close fails, but what?
Andy Lester though we should do something with the return value, otherwise it means that "a close will never fail". Dave Mitchell, ever the voice of reason, pointed out that when the file handle is a lexical, and is being closed because the lexical is going out of scope, as in
{
open my $fh, ....;
}
# implicit close here
there is no possible way, short of dying, to tell the program that Something Bad happened.
http://xrl.us/hxtb
Alan Burlison was Warnocked over a question of how to get MakeMaker
deal with the tricky symlinks necessary to build DBD::Oracle,
where the symlink hackery would let one build a perl that speaks
to Oracle without a Oracle client install, thereby saving 400
megabytes of disk space.
He must have figured a way forward, because he later wrote back to
say that his compile of DBD::Oracle on Solaris 10 had uncovered
horrors lurking in the dim, rarely visited corners of the perl
source. It's basically a problem with %ENV. Perl_my_setenv
can actually reach out and directly manipulate the memory pointed
to by the environ pointer that the system passes to the perl
executable.
The new, improved method Solaris 10 employs for managing the
environment breaks an assumption long held by the perl5 porters.
And results in bizarre "Out of memory" errors during perl_destruct.
After having looked at the code that deals with getenv and
putenv, and having considered it to be "pretty vile", Alan asked
what it was all supposed to achieve.
After discussing the issue a while with Steve Peters, Steve
suddenly made the connection between what Alan was seeing and what
happens on the Cygwin platform. In a nutshell, does delete $ENV{foo}
just delete the value, but leave foo= in the environment
table, or does it delete the key and value altogether?
Andy Dougherty pointed out that much of the code dates back to when no standards existed for these matters. These days, however, it's probably safe to assume that the vendor's library is sane and should be used by default, and only invoke work-arounds when there is good reason to.
All summed up in bugs #37376 and #37377.
Managing synlinks with MakeMaker http://xrl.us/hxtc
The stench begins http://xrl.us/hxtd
The link with Cygwin http://xrl.us/hxte
The source code for Solaris's getenv http://cvs.opensolaris.org/source/xref/usr/src/lib/libc/port/gen/getenv.c
Michael Schwern released a new version of Test::Simple. The thread
on p5p is rather succinct. To get the full picture, you had to be
following perl-qa. Over there, Adam Kennedy noticed that the
September 23 release of Test::Simple/More broke
Test::Builder::Tester (that scraped STDERR to pick up the failure
messages). Which in turn is used by about 26 other testing modules.
So Adam asked Michael to back out the changes until T::B::T
could be updated.
Given the number of fixes that went into that release, Michael quite
naturally refused to back the changes out. The main problem is that
people had ample time to try out the betas to see if the
changes would cause problems. Unfortunately, Mark Fowler, author of
T::B::T was holed up in bed, nursing a ferocious cold.
So the new release of Test::Simple embraces and extends T::B::T
and includes it as its own, with the amended screen scraping in place.
I believe that in times like these, one is supposed to say:
You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it.
In the meantime, imminent meltdown of the global infrastructure for testing Perl modules was averted.
I suppose the moral of the story is that if you write a module that has a another module as a prerequisite, you should make sure you are aware of all changes made to the prerequisite, whether it be by watching new beta uploads (or writing a program to do the watching for you), or subscribing to the right mailing list or something else. Doubly so if you're walking around with a shot-gun in the lounge, or whatever the saying is. Nothing is ever carved in stone.
The thread on perl-qa http://www.mail-archive.com/perl-qa@perl.org/msg04740.html
Test/Simple/More/Builder is released http://xrl.us/hxtf
The Perl5 bug summary has 1512 open items
http://xrl.us/hxtg
Sébastien Aperghis-Tramoni asked for an addition to Devel::PPPort,
to make life easier for Net::Pcap, for much the same thing that Ken
Williams is already doing in Cwd.xs.
http://xrl.us/hxth
Paul D. Lalli found that:
my $i;
my @a = (1..10);
my $last = $#{@a};
produced the message "Bizarre copy of ARRAY in leave". Then again, if I'm not mistaken, I think that code wants:
my $last = $#a;
Bug #37350 http://xrl.us/hxti
H.Merijn Brand added -C as an allowed flag to the PERL5OPT environment
variable. (Which you can use to set command-line switches for all invocations
of perl, so one would hope that -u is not allowed).
http://xrl.us/hxtj
Andy Lester checked in with some more const work, and cleaned up printf.
Applied by H.Merijn.
http://xrl.us/hxtk
Rajarshi Das patched utf8.c on EBCDIC platforms. Sadahiro Tomoyuki asked for regression tests, and provided four, two that currently fail and two that pass. Rajarshi's fix should ensure that all four pass. Rajarshi ran the tests and all passed.
http://xrl.us/hxtm
Mohammad Yaseen asked about ext/B/t/optree_specials.t when running on z/OS (an EBCDIC platform), and was Warnocked.
http://xrl.us/hxtn
"jafoc" said (in #37355) that bad things happen when you try to run
perl -c -e 'my $xyz; undef xyz'
on perl 5.6.1. Yves "demerphq" Orton replied that it was fixed in 5.8.
http://xrl.us/hxto
Following on from Michael Schwern's announcement of EU::MM 6.30_01,
Peter Prymmer came up with some patches for VMS:
http://xrl.us/hxtp
Tassilo von Parseval reported in #36875 (August 2005) that for perls between 5.5.4 (did he not mean 5.8.4?) and 5.9.3 that
perl -we 'print lc(undef)'
would not emit a warning about uninitialised values. Steve Peters confirmed the bug, and had a preliminary patch beginning to take shape.
http://xrl.us/hxtq
In #37384, Dan Dascalescu reported that
C<perl -we '$a=qr//; $b=qr//x; 1 =~ ($a|$b)'>
fails with a 'Sequence (?}...) not recognized in regex'. Dave Mitchell pointed out that bitwise-oring two string is liable to make for a pretty odd regular expression...
This summary was written by David Landgren.
Information concerning bugs referenced in this summary (as #nnnnn) may be viewed at http://rt.perl.org/rt3/Ticket/Display.html?id=nnnnn
Information concerning patches to maint or blead referenced in this summary (as #nnnnn) may be viewed at http://public.activestate.com/cgi-bin/perlbrowse?patch=nnnnn
Weekly summaries are published on http://use.perl.org/ and posted on a mailing list, (subscription: perl5-summary-subscribe@perl.org ). The archive is at http://dev.perl.org/perl5/list-summaries/ . Corrections and comments are welcome.
If you found this summary useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl.
Let me say that the so-called 'screen-scraping' idea for T::B::T was entirely the author of Test::Builder's idea and uses only documented interfaces. There is no 'breaking inside' or 'peeking' going on. If you can think of any better way to determine if the diagnostics that are output from your new testing module matches what you expected to output than capturing the output and comparing it, then I'd like to hear it.I believe that in times like these, one is supposed to say: You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it.
I suppose the moral of the story is that if you write a module that has a another module as a prerequisite, you should make sure you are aware of all changes made to the prerequisite, whether it be by watching new beta uploads (or writing a program to do the watching for you), or subscribing to the right mailing list or something else. Doubly so if you're walking around with a shot-gun in the lounge, or whatever the saying is. Nothing is ever carved in stone.
Or I could not bother releasing software to CPAN in the first place at all - you're asking module authors to do a *lot* of work here. There's no automatic mechanism provided as a way to notify me if a new beta version of a module is uploaded. Quite frankly I don't have time to read all the Perl mailing lists for every single prerequisite of every module I have uploaded. Who does? And as for writing a tool to do automatic monitoring, it's a nice idea, but completely impractical and not at all core to what I'm trying to do - which is develop the software I'm trying to develop not spend time writing tools that monitors the current contents of CPAN.
It's the expectation that I should jump right then and there because someone else has released something that's broken a third party's code that I find annoying. It's neither really my problem or a problem of my making. I'm willing to take the time 'fix' my modules because I like to think of myself as a nice guy, but unless people are prepared to realise that Perl developers have their own priorities (or at least, their employer's) then there's going to continue to be a lot of ruffled feathers.
Maybe I'm not up to maintaining a CPAN module. Maybe I'll not bother uploading modules in the future to avoid the grief.
Excuse me while I pick up all my toys and put them back in the pram.
Re:On the Test::Builder::Tester lynch mob
schwern on 2005-10-11T22:09:20
The method of grabbing the screen output is documented, but exactly what gets output is not. The way I put it was:I believe that in times like these, one is supposed to say: You are wicked
and wrong to have broken inside and peeked at the implementation and then
relied upon it.
Let me say that the so-called 'screen-scraping' idea for T::B::T was entirely the author of Test::Builder's idea [perl.org] and uses only documented interfaces.
You are wicked and wrong to have scraped the screen and then relied upon it.
As Mark points out, it was my idea. I'm the one who is wicked and wrong. Or, at least, the one passing out delicious, poison apples. Mark, sorry if you came off as the bad guy here.
Further egg on my face: during the alpha phase, Test::Warn's tests were failing when I tried to install it as a prereq for AI::Prolog. I dismissed it at the time. It was TBT failing and I ignored it.
After calming down and getting past the cries of "Roll back Test::More!" I realized there was a of collective breakdown on perl-qa. Nobody spotted this problem. Despite the long discussions on about changing the Test::Builder failure diagnostics, nobody thought to look at TBT. None of the authors of testing modules who rely on TBT tried out the two Test::More alphas which contained the change. I didn't think to, either. Nobody realized TBT was doing anything wrong until it broke and its been out for over a year.
The change was discussed, the alphas were publicized on several lists, there was ample lead time, TBT is used directly by many very active modules, the change in question isn't even a documented feature. It caused a massive cascade failure and even then it wasn't noticed for over a week. How could so many people miss such a widespread failure? What can be done to fix that?
Re:On the Test::Builder::Tester lynch mob
2shortplanks on 2005-10-12T09:15:34
Let's face facts, we're busy, busy people. Rafael's point about monitoring releases (particularly beta ones) is a good one. I should have been doing that. But it's very hard - I don't have time to read the perl-qa list every week, and it's sometimes months between when I blast though each of my mailing lists. Why don't we look at what steps can be taken to make it easier to monitor what releases are coming out?How could so many people miss such a widespread failure? What can be done to fix that?I like the point about an automated tool, but I'm certainly not going to write one. Or I might, but if I do then I should find some way to allow other people to use it. And I don't mean just by giving them the tool to run too. I mean making some kind of service available.
It's about time we produced some fine-grained notification method for new module releases. If this was a couple of years ago, I'd have said we could let people register email addresses. This day and age I'd say we need to have a system that people can subscribe to releases via their RSS readers.