New WWW::Mechanize and Test::WWW::Mechanize spiffiness

petdance on 2007-10-31T17:13:56

For those of you using Mech for your testing of your website:

    $agent->content_contains( qr/\QEnter keyword(s)/ )
        or $agent->dump_all( \*STDERR );

not ok 14 - Content contains '(?-xism:Enter\ keyword\(s\))' # Failed test 'Content contains '(?-xism:Enter\ keyword\(s\))'' # at t/simple-search.t line 31. # searched: "\x{0a}\x{0a}TitleTales™ (checkbox) [*/off|on/Include Out of Print / Please Order Direct Titles]


No longer do you have to do a $mech->save_content() and then run mech-dump on it. How has it taken me so long to put this stuff in there?


See Also: WWW::Mechanize::Plugin::Display

markjugg on 2007-10-31T17:40:31

Thanks Andy.

A related (but slightly different) tool is WWW::Mechanize::Plugin::Display.

Default Output Stream

Smylers on 2007-10-31T17:56:57

Do you think that will be the canonical use of dump_all?

If so, would it make more sense for the default output stream to be STDERR rather than STDOUT, to save having to specify it every time?

Re:Default Output Stream

petdance on 2007-10-31T18:06:55

I don't know what the canonical use will be, so I defaulted to STDOUT.

I hate to be a spoil sport, but...

Alias on 2007-10-31T23:41:21

... when are you going to get red of whatever test server you are using for the test scripts that keeps failing and causing mech to be an unreliable dependency?

I've started to migrate away from mechanize back to raw lwp in a few applications, simply because mech has become too unreliable.

Re:I hate to be a spoil sport, but...

petdance on 2007-11-01T00:31:33

I didn't know it was a problem until you mentioned it yesterday.

I'm open to alternatives.

Re:I hate to be a spoil sport, but...

mw487 on 2007-11-01T13:42:27

In the past I think it failed most of the time on cygwin, due to the forking necessary for the server I believe, and I pretty much went back to LWP in spite of how nice mech seems (although mech can be an install festival).

Maybe the first thing you could do would be test that mech tests still fails on cygwin (untested Mech 1.32, fails 1.3102, untested 1.3101, fail:pass 5:1 1.30, etc.).

And then test the many flavors of http servers that might work. Just run there tests on Cygwin or review their results on the test-result server.

If you have no cygwin, I could run things for you.

I have the latest cygwin, and cygwin supplied perl, and mech hangs on t/local/back after reporting test 12 and failing 5,6,9...

t/local/back.............1..38
ok 1 - use WWW::Mechanize;
ok 2 - The object isa WWW::Mechanize
ok 3 - this $mech starts with a cookie jar
# Started http://localhost:1748/
ok 4 - The object isa LocalServer

not ok 5 - Fetched OK
# Failed test 'Fetched OK'
# at t/local/back.t line 52.

not ok 6 - Followed OK
# Failed test 'Followed OK'
# at t/local/back.t line 58.
ok 7 - Did the base get set back?

# Failed test 'Followed OK'
ok 8 - Title set back?
# at t/local/back.t line 65.
not ok 9 - Followed OK
ok 10 - Did the base get set back?No form defined at t/local/back.t line 72

ok 11 - Title set back?
ok 12 - Pre-search check

[hangs]

[ctrl c]

$ perl -V
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
    Platform:
        osname=cygwin, osvers=1.5.24(0.15642), archname=cygwin-thread-multi-64int
        uname='cygwin_nt-5.1 reini 1.5.24(0.15642) 2007-01-31 10:57 i686 cygwin '
        config_args='-de -Dmksymlinks -Duse64bitint -Dusethreads -Uusemymalloc -Dopt
imize=-O3 -Dman3ext=3pm -Dusesitecustomize -Dusedevel'
        hint=recommended, useposix=true, d_sigaction=define
        usethreads=define use5005threads=undef useithreads=define usemultiplicity=de
fine
        useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
        use64bitint=define use64bitall=undef uselongdouble=undef
        usemymalloc=n, bincompat5005=undef
    Compiler:
        cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -Wdecl
aration-after-statement',
        optimize='-O3',
        cppflags='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -Wdeclaration-af
ter-statement'
        ccversion='', gccversion='3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
', gccosandvers=''
        intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
        d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
        ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lsee
ksize=8
        alignbytes=8, prototype=define
    Linker and Libraries:
        ld='ld2', ldflags =' -s -L/usr/local/lib'
        libpth=/usr/local/lib /usr/lib /lib
        libs=-lgdbm -ldb -ldl -lcrypt -lgdbm_compat
        perllibs=-ldl -lcrypt -lgdbm_compat
        libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=libperl.a
        gnulibc_version=''
    Dynamic Linking:
        dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' -s'
        cccdlflags=' ', lddlflags=' -s -L/usr/local/lib'

Characteristics of this binary (from libperl):
    Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                                                PERL_MALLOC_WRAP PERL_USE_SAFE_PUTENV
                                                USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
                                                USE_PERLIO USE_REENTRANT_API USE_SITECUSTOMIZE
    Locally applied patches:
                CYG01 - hints.cygwin.sh ldflags -s
                CYG02 - lib-ExtUtils-Embed insensitive against leading \s
                CYG03 - lib-Test-Harness-Straps $ENV{PERL5LIB} = ''
                CYG04 - major.version.cygwin.sh cygperl-5_8.dll and not cygperl-5_8_x.dl
l
                CYG05 - add Win32CORE to core
                CYG07 - File-Spec-Cygwin-TMPDIR.patch
                Bug#38628 - allow legacy Cwd->cwd()
                Bug#40103 - File-Spec-case_tolerant.patch from 5.9.5
    Built under cygwin
    Compiled at Jul 8 2007 19:12:08
    %ENV:
        PERL_MODULE_AUTHOR="xxx"
        CYGWIN=""
    @INC: /usr/lib/perl5/5.8/cygwin /usr/lib/perl5/5.8 /usr/lib/perl5/site_perl/5.8/cygwin /usr/lib/perl5/site_perl/5.8 /usr/lib/perl5/site_perl/5.8 /usr/lib/perl5/vendor_perl/5.8/cygwin /usr/lib/perl5/vendor_perl/5.8 /usr/lib/perl5/vendor_perl/5.8
        .

Re:I hate to be a spoil sport, but...

petdance on 2007-11-01T14:39:05

I can't test it on Cygwin. Regardless, my question still stands: What should I run my tests against instead?

Re:I hate to be a spoil sport, but...

mw487 on 2007-11-01T18:47:06

I think you could fix the problems with your testing if you used a cross-platform, CPAN-tested local httpd rather than the included LocalServer.pm . Maybe you could pick a CPAN httpd that passes tests on cygwin and ActiveState and Linux etc.

NOT HTTP-Server-Simple, NOT HTTP-Server-Brick, but get twisted with POE-Component-Server-SimpleHTTP, or take what is probably the easy way with Net::HTTPServer notably 1.1.1, a distribution which shows only one fail on CPAN-Testers for about 60 submissions on various versions. I just verified that 1.1.1 completes its 141 tests on my up-to-date cygwin, and someone else verified MSWin32.

Re:I hate to be a spoil sport, but...

petdance on 2007-11-01T18:52:55

Can you help out on this at all? Also, what's wrong with HTTP::Server::Simple?

Re:I hate to be a spoil sport, but...

mw487 on 2007-11-01T20:11:28

I can help out. I already put Net::HTTPServer on my cygwin and ActiveState Perl MSWin32.

For starters, I can pick up a distribution using Net::HTTPServer for the back.t test, passing on your platform, from a url you supply and run the tests on cygwin and ActiveState MSWin32. That would mean you change your test to use Net::HTTPServer over LocalServer.pm.

If you want me to do make the changes, I will need to learn some of your code first, and I may bail, but likely will get to it in time. I like mech, but am mostly on Win32 and hate to force.

I do not suggest HTTP::Server::Simple because of the test results on MSWin32 and cygwin.

http://tinyurl.com/38c495

http://cpantesters.perl.org/show/HTTP-Server-Simple.html#HTTP-Server-Simple-0.27

because:

0.28_01: FAIL 547183 5.8.8 on Cygwin 1.5.24(0.15642) (cygwin-thread-multi-64int)

0.27: FAIL 725025 5.8.8 on MSWin32 5.1 (MSWin32-x86-multi-thread)

FAIL 522118 5.8.8 on Cygwin 1.5.24(0.15642) (cygwin-thread-multi-64int)

there are some passes for Win32, but many more fails.

I wish the preview button on use.perl.com were not broken- hope this looks okay.

Re:I hate to be a spoil sport, but...

petdance on 2007-11-01T20:35:14

I'd be glad to have your help. The Mech code lives on Google Code. I'd also really like to be able to get rid of the cut 'n' paste in the t/ directory to support it all.

What can I do to help you?

Re:I hate to be a spoil sport, but...

mw487 on 2007-11-02T20:06:24

Did you get my emails to andy@petdance.com? I am mw487@yahoo.com. This is getting into the weeds.

In t/local/*.t, change:

  delete @ENV{ qw( http_proxy HTTP_PROXY ) };

to

foreach my $proxy (grep {/http_proxy/i} keys %ENV ) {delete $ENV{$proxy}}

because believe it or not, Win XP takes my definition of environmental variable 'http_proxy' and kindly gives me 'HTTP_PROXY' and 'HTTP_proxy'. Whereas for 'ftp_proxy' I only seem to get 'FTP_PROXY' as a bonus.

That change lets me pass t/local/back.t .

Re:I hate to be a spoil sport, but...

mw487 on 2007-11-02T21:17:15

Actually, the HTTP_proxy problem seems to be in cygwin perl only, not due to XP, because ActiveStatePerl does not do show the problem. Guess I should file a bug report somewhere...

C:/ActiveStatePerl/bin/perl -e 'foreach my $proxy (grep {/_proxy/i} keys %ENV) {print "\$ENV{$proxy} => $ENV{$proxy} \n" ; delete $ENV{$proxy}; } '

If I leave my http_proxy and set my environment no_proxy to localhost, I can pass almost every test, except that HTTP:Daemon is a smart ass and uses my machine name instead of localhost. I added my machine name to no_proxy, but there is still a subtle problem with having even HTTP_proxy and no_proxy.

Because then in failure.t, you connect to Bad hostname \'blahblahblah.xx-only-testing, which of course goes to my HTTP_proxy, which nicely informs mech in HTML that it can not find the host, so it fails test 13 "is not html", because it IS HTML.

So, I still favor going with /http_proxy/i .

Re:I hate to be a spoil sport, but...

petdance on 2007-11-02T21:27:24

Can you please summarize to http://code.google.com/p/www-mechanize/issues/detail?id=8 and carry on commentary there?

Re:I hate to be a spoil sport, but...

Aristotle on 2007-11-02T23:34:05

No need to write C in Perl:

delete @ENV{ grep { /http_proxy/i } keys %ENV };