Yesterday we noticed some slowness with HTTP on a machine at $work, and I checked the Apache log. It was - to say the least - larger than expected. tail -f default.access_log showed someone is accessing the machine (through IP, not website) quite often, producing 404 errors and using bad user agents. Stuff like googlebawt and the sort. So, we got an attack on the server through Apache, and they are trying to DDOS.
Fortunately, I'm blessed (and I'm not being cynical about this) with an environment of highly skilled programmers. My boss is an OpenBSD and Gentoo fanatic (who could blame him, really?) to just stress the point.
I started working on blocking the originating attack using mod_rewrite on Apache according to the user agent, when my boss said we should focus on iptables block because perhaps it's not coming from a large range of IPs. I wrote a short oneliner in Perl to put the IPs in a hash with a count of how many times they appear. Apparently we were dealing with roughly 100 or so different IPs generating 400Kbit transfer. We've handled more than 10000 times as much on more serious attacks. This was feather-weight.
We created another chain in the iptables and I wrote a short Perl script (originally oneliner, but I wanted it structured in a file) that blocks with iptables anyone that generates more than $limit 404 errors, which I put in the cron and after 3 seconds, the attack was blocked, almost completely. I left it running all night and the log shows it blocked only an additional 22 IPs over the night.
Here is the script, in case anyone's interested. For some odd reason List::MoreUtils's none didn't work for me. Sometimes I get confused by it and it was just faster to do it without it, I know.
reverse)
#!/usr/bin/perl
use strict;
use warnings;
use File::Slurp;
# these next 3 should be absolute paths
my $log = 'default.access_log';
my $dropped = 'dropped';
my $my_logfile = 'drop.log';
my $limit = 150;
my $real = 1;
my $read_error = 0;
my %by_ip = ();
my %by_count = ();
if ( $ARGV[0] ) {
$real = 0;
}
foreach my $filename ( $log, $dropped ) {
if ( ! -r $filename ) {
print "File '$filename' doesn't exist or is unreadable!\n";
$read_error++;
}
}
$read_error && die "We have file read errors\n";
open my $fh, '<', $log or die "can't open file '$log': $!\n";
foreach my $line ( <$fh> ) {
if ( $line =~ /HTTP\/1\.1\"\s404/ ) {
( my $ip, undef ) = split /\s/, $line;
$ip && $by_ip{$ip}++;
}
}
close $fh;
# not necessary, but useful to see best targets
# %by_count = reverse %by_ip;
while ( my ( $ip, $count ) = each %by_ip ) {
if ($count > $limit ) {
# cron does not have PATH variable set
my $cmd = "/sbin/iptables -I ddos -s $ip -j DROP";
my $found = 0;
my @dropped_ips = map { chomp; $_ } read_file($dropped);
foreach my $dropped_ip (@dropped_ips) {
if ( $dropped_ip eq $ip ) {
$found++;
last;
}
}
if ( !$found ) {
my $cur_time = scalar localtime time;
my $msg = "($cur_time) $cmd [number of times: $count]\n";
if ($real) {
append_file( $my_logfile, $msg );
system $cmd;
append_file( $dropped, "$ip\n" );
} else {
print $msg;
}
}
}
}
Notes:
List::MoreUtils's noneFile::Slurp slurps the entire file if I do foreach my $line ( read_file($filename) ) and if not, use itGetopt::LongRegexp::Assemble for multiple regexp tests> I should have compiled the regexp with qr// at start and then use the compiled version
If you're referring to the "/HTTP\/1\.1\"\s404/" in the loop - perl is smart enough to only compile it once, as it doesn't contain any variables that would change it.
You can test this be running
$ perl -Mre=debug -le 'for (1..2) { $_ =~
and watching the output
Re:qr not necessary
xsawyerx on 2009-03-05T15:48:03
Interesting.
Thank you!
There is IPTables::libiptc which is a more modern replacement to IPTables::IPv4 (no longer maintained). The risk with these modules is that the kernel's interface to libiptc changes (as it's not a proper published API).
It's also a good idea to use some kind of privilege separation when needing to do operations as root. For example you watch the Apache logs as an unprivileged user, then to make a change to Netfilter, send an RPC command to another process using a defined and safe interface. The other process runs as root but has the minimal code required to do its job, i.e. few bugs. We use my RPC::Serialized module at work, to do this.