Leaky buckets

acme on 2007-05-22T10:47:33

My buckets are leaky! If you ever have to do anything like rate limiting or throttling you should look at the token bucket algorithm. Handily CPAN has Algorithm::TokenBucket (which is similar) to make it easy for you. I recently wanted to throttle Net::SFTP::Foreign and ended up with something like this:

use Algorithm::TokenBucket;
use Net::SFTP::Foreign::Compat;
use Time::HiRes qw(sleep);

my $mbps   = 0.5;
my $bps    = $mbps * 1024 * 1024 / 8;
my $bucket = Algorithm::TokenBucket->new( $bps, 32768 );

...

        my $last_offset = 0;
        $sftp->get(
            $remote_filename,
            $filename,
            sub {
                my ( $sftp, $data, $offset, $size ) = @_;
                my $bytes = $offset - $last_offset;
                $last_offset = $offset;
                sleep 0.01 until $bucket->conform($bytes);
                $bucket->count($bytes);
            }
        );