I think I have discovered something horrible about Expect.pm - I can only hope that I am wrong. When Expect spawns a process, the parent doesn't bother to wait for the child to complete. It is only by intercepting command line (i.e. tty) output, in conjunction with a timeout value, that it continues the "session".
But what if nothing returns on the command line? The parent dies, killing any child that may be running. This is bad. This is very bad.
When I set up my Net::SCP::Expect module, I set up a 1 second (default) timeout to check for any error messages. I also used a 'soft_close()' followed by a 'hard_close()' in the event that the former failed. The significance of these things will be made clear shortly.
In testing, I never tried copying any particularly large files and so I never had any problems. However, I recently received an email by someone using my module that he was having trouble copying large files. It seemed that he was only getting partial copies. So, I tried it myself - he's right. When I tried copying a rather large log file over, only a portion of it was copied before the program exited.
The next step was to just use the Expect module on its own. Same problem - the program dies before the whole file is copied over. What do I discover? That the only thing keeping the parent going at all is the timeout I used for error checking and the lag caused by using the 'soft_close()' method (which always seems to slowly fail). Once I removed those, even *less* of the test file was copied over. However, this small timeout plus the lag was always enough time to copy the small files I used in testing, so that's why I never noticed until now.
So far as I can tell, if Expect doesn't get anything back on the command line (and it shouldn't since I'm running scp in quiet mode), the parent exits and kills and child process that may be running. The only workaround is to set a high 'timeout_err' value in my module, so that it has enough time to copy before the program exits.
Ouch. Big fucking ouch. An email from Roland Giersig wasn't of much help. So, what do I do now in the event there's no way to deal with this? Do I rip out the Expect module completely and replace it with one of the Term modules? I may very well have to.
UPDATE: jdavidb was kind enough to provide a fix. Hooray! Thanks jdavidb!
I think this behavior is similar to what you would have had under TCL/Expect. Not that that's a good reason.
Can't you expect eof? Under TCL/Expect, if you wanted to run a command that produced no output, you would do an expect eof with a long (or -1) timeout to wait until it ended.
Are you using the scp command-line program? If so, doesn't it produce a progress bar that would give you some output to expect()? At first I didn't understand why you were using Expect, but then I read the docs and I see it's for supplying passwords. I'm sure Roland's suggestion was to use keys, and I'm sure you already have a reason you don't want to (or can't).
Still don't understand why you have to do soft_close() and hard_close(). Maybe if I reread what you said again.
I do think $exp->expect($timeout, 'eof') will do what you need.
Re:Thoughts
djberg96 on 2002-06-05T15:21:50
Can't you expect eof? Under TCL/Expect, if you wanted to run a command that produced no output, you would do an expect eof with a long (or -1) timeout to wait until it ended.Ooooh - good idea. I'll give it a shot.
doesn't it produce a progress bar that would give you some output to expect()?
Not in quiet mode. Normally you would get a '#' sign, but I wasn't sure how portable, or reliable, it was to check for '#'.
Still don't understand why you have to do soft_close() and hard_close().
I don't have to. The docs indicate that soft_close() was the polite way to close and hard_close() was the impolite way so my implementation was arbitrary based on that. I should probably just rip out the soft_close() altogether since it doesn't seem to work at all (on my system anyway).
I do think $exp->expect($timeout, 'eof') will do what you need
I will definitely give it a shot. Thanks!
Re:Thoughts
jdavidb on 2002-06-05T15:32:40
Still don't understand why you have to do soft_close() and hard_close().
I don't have to. The docs indicate that soft_close() was the polite way to close and hard_close() was the impolite way so my implementation was arbitrary based on that.
No, I mean I don't understand why you call those routines at all. Usually you would only have to call them if you were ending the command early. Once you get to the point of eof either close is redundant, I think.
Re:Thoughts
djberg96 on 2002-06-05T15:46:16
Didn't work. Strangely, it seems to be getting a blank char sequence followed by a "\r\n" sequence. Here's a snippet of the verbose output:Even if I intentionally try to match a "\r\n" sequence, all I can do is a "sleep $timeout" or something. You can't stick an 'expect' call in a while loop - it only checks once in case you were thinking of something like:Sending 'password\n' to spawn id(3)
Expect::print('Expect=GLOB(0x2fc558)', 'password^J') called at./scpTest3.pl line 21
Starting EXPECT pattern matching...
Expect::expect('Expect=GLOB(0x2fc558)', 1, 'eof') called at./scpTest3.pl line 31
spawn id(3): list of patterns:
#1: -ex `eof'
spawn id(3): Does ` '
match:
pattern #1: -ex `eof'? No.
spawn id(3): Does ` \r\n'
match:
pattern #1: -ex `eof'? No.
Closing spawn id(3).
Expect::hard_close('Expect=GLOB(0x2fc558)') called at./scpTest3.pl line 34
spawn id(3) closed.while($scp->expect(1,"\r\n")){ sleep 1 } # Doesn't workRe:Thoughts
jdavidb on 2002-06-05T16:10:36
Okay, this is goofy, but it appears you can't just
$exp->expect($timeout, 'eof')
Otherwise, it tries to match the literal string 'eof' (hence the -ex in your verbose output). You actually have to use the arrayref syntax to expect(). The arrayref syntax is
$exp->expect($timeout, [pattern => code], [pattern => code],
...) So you can do
$exp->expect($timeout, ['eof' => sub { exit }])
I tried leaving out the => sub { exit } and it didn't seem to work, although I seem to have working code elsewhere that functions that way (perhaps that code isn't working after all...). In any case you could do sub { } if you wanted.
Re:Thoughts
jdavidb on 2002-06-05T16:21:46
Ah! Found out that I only need the => sub { } on special things like eof because they use special code that didn't take the possibility of not specifying a coderef into account.
Re:Thoughts
djberg96 on 2002-06-05T16:44:56
Yeah, that's better, but it's still getting "\r\n" and not eof anyway, so it's a moot point. Besides, I still have to deal with it via a timeout value (right?) so no matter what, your only option currently is to *guess* how much time you need to complete the operation.The fundamental flaw with Expect.pm (and perhaps expect in general?) is that it kills its own children before letting them finish. Perhaps there was a reason for this, or perhaps they never counted on anyone performing an op that didn't return anything to the terminal. Either way, I'm gonna have to decide what to do.
:( Re:Thoughts
jdavidb on 2002-06-05T17:03:45
You can specify an unlimited timeout. It's hidden in the docs somewhere. Use undef instead of a value. In the original TCL/Expect, it was -1. (Had trouble finding it in the docs back then, too, as some of my old programs attest.)
Here's my code to scp a file from my machine to its own
/tmp directory: #!/usr/local/bin/perl5.6.1
use warnings;
use strict;
our $password = "hello";
use Expect;
my $exp = Expect->spawn("scp file.xls minako:/tmp");
$exp->expect(2, "Password: ");
$exp->send("$password\r");
$exp->expect(2, ['eof' => sub { }]);
Is this fundamentally different from what you're trying to do?
I don't think it's a fundamental flaw; if you want to wait until the end, you specify it explicitly with eof. Not sure why "\r\n" makes a difference. If you want to match "Password: " but are worried about the case of the letter, you can match "ssword: ". You don't have to match everything, just the last thing you want.
Boy, you go to great lengths to get someone to install your module.
I installed Net::SCP::Expect and tinkered with it a bit. I was able to reliably duplicate the problem by attempting to scp the largest file on my machine to my
--- Expect.pm.orig Mon Apr 29 15:14:48 2002
+++ Expect.pm Wed Jun 5 12:28:00 2002
@@ -37,7 +37,7 @@
_auto_yes => $arg{auto_yes} || 0,
_timeout => $arg{timeout} || 10,
_timeout_auto => $arg{timeout_auto} || 1,
- _timeout_err => $arg{timeout_err} || 1,
+ _timeout_err => $arg{timeout_err} || undef,
_no_check => $arg{no_check} || 0,
};
@@ -228,7 +228,10 @@
}
}
],
+ ['eof' => sub { }],
);
+ } else {
+ $scp->expect($timeout_err, ['eof' => sub { }]);
}
if($verbose){ print $scp->after(),"\n" }
It seemed to be necessary to set timeout_err to undef within Net::SCP::Expect.pm; I tried to do it with my constructor but must not have understood how to do it. You'll probably not want to change it in the module itself, but I was in a hurry.
Here's my code that (after those changes) successfully transferred a large file (35M):
#!/usr/local/bin/perl5.6.1
use warnings;
use strict;
use Net::SCP::Expect;
my $user = "me";
my $password = "xyzzy";
my $file = "/home/jb1949/Copland,_Aaron/Appalachian_Spring.mp3.bz2";
my $destdir = "minako:/tmp";
my $scp = Net::SCP::Expect->new(timeout_err => undef);
$scp->login($user, $password);
$scp->scp($file, $destdir);
Thoughts upon hitting "Preview": posting a patch in slash isn't very useful.
Re:Fix
djberg96 on 2002-06-05T18:30:46
Sweet, sweet, SWEET!!!! Works like a charm. Setting the timeout_err to undef isn't going to work because of the " || 1 " in the constructor. I changed it to " || undef ". Perhaps, I should allow a literal string 'undef' for that option, and then deal with it in the constructor as appropriate.Thank you very much. Should I be irritated that I ultimately received the answer from you and not Roland? Hmm...nevermind. I may end up in the same boat some day. That, or he thought I was an annoying nit and just wanted to torture me.
If you want to match "Password: " but are worried about the case of the letter, you can match "ssword: ". You don't have to match everything, just the last thing you want.
Ahhh, but did you know that some versions of scp ask for your "password" and others ask for your "passphrase"?
:) Oh - please send me an email with your name so I can credit you in the changelog. If there's a way to view someone's profile here on use.perl I haven't figured it out yet.
Re:Fix
jdavidb on 2002-06-05T18:48:20
Should I be irritated that I ultimately received the answer from you and not Roland?
Speaking as an active member of the expectperl mailing list, Roland sees a lot of FAQs asked many, many times. We also have a lot of people who are trying to learn Expect.pm before they learn Perl! (It doesn't help that some have never seen TCL/Expect, either.) It seems like every week there's someone trying to automate ssh in a way that indicates they would do better to try a different method than Expect (such as system()). He's not unresponsive, just tired of being repetitive.
:) [BTW, you have joined the expectperl mailing list, right?] Ahhh, but did you know that some versions of scp ask for your "password" and others ask for your "passphrase"?
:) qr/pass(word|phrase)/i ; it's a better solution to the problem I posed, anyway.
J. David Blackstone profile
Re:Oh?
jdavidb on 2002-06-06T18:40:57
Perhaps a couple of FAQ entries would be in order. I'll draft and send to the mailing list.
- My spawned process doesn't produce any output, and Expect.pm doesn't wait for it to end.
- There's no way to reliably decide how long to set my timeouts; can't I make it unlimited?
Re:Oh?
RGiersig on 2002-06-07T09:06:36
Sure, if we could only get people to read them...;-) Thanks, I'll put them in and also take a look at the 'eof' issue.
Austin Shutz (the original author of Expect.pm, I think) offers this other way to do it:
$exp->expect(undef);This will basically just wait until the command is done; expect() will return when it gets the eof. If you really needed to explicitly see the eof at that point, you could check the error string:
if ($exp->error() =~