The Dashboard shows visitors and robots

Security Etudes : #100 : Security : IPs

Who is hacking

Plenty of crooks and idiots out there

• For last month some of my websites had been attacked from various places
• I run software on non-standard ports, but it is never a defense from deliberate attack
• Turns out that cloud providers out there don't really care much about crooks using their servers to hack into other people servers
• Last month I've been attacked from GoGrid, HE, Amazon AWS, Webair (not counting Russian and Chineese IPs)
• There are several simple technical steps that ISPs could use to put hackers out of business, and maybe they will do that some day
• In the modern world - if you want to defend your server from hackers - you're on your own

This is what I do now (script does it, on autopilot)

• Look who is trying to hack into my server
• It is as simple as zcat /var/log/* | ips.pl
• Block their IPs with iptables
• This would not protect from DDOS attacks
• Your ISP can protect from DDOS - with a big honking loadbalancer

Why no whitelisting?

• Does not work on port 80
• I generally like to know who exactly is trying to hack into my servers and trace it to the source (hi, little company from Chicago)

Update (Apr 06. 2013)

There is better way.

• Split traffic into 2 groups
• First group - http
• Second group - 'everything else'
• Deal with http attacks in one way and deal with 'everything else' - in another way (based on tcpdump)

Update (May 02. 2013)

https://github.com/kickstarter/rack-attack - looks like a good idea to deal with http group of abusers

Update (Jul 28. 2013)

AutoBan - self-learning intrusion detection system in 100 lines of Perl

March 01. 2013

Background

• Sony+Hackers+AWS

  Amazon.com's Web Services were used by hackers in the April attack against Sony's online entertainment services, according to a Bloomberg ...

• Detecting TOR Communication in Network Traffic

  The anonymity network Tor is often misused by hackers and criminals in order to remotely control hacked computers. In this blog post we explain why Tor is so well suited for such malicious purposes, but also how incident responders can detect Tor traffic in their networks.

• SNORT

  Snort® is an open source network intrusion prevention and detection system (IDS/IPS) developed by Sourcefire. Combining the benefits of signature, protocol, and anomaly-based inspection, Snort is the most widely deployed IDS/IPS technology worldwide. With millions of downloads and nearly 400,000 registered users, Snort has become the de facto standard for IPS.

• Rack::Attack: protection from abusive clients

  If you've looked at web server logs, you know there are some weird clients out there. Malicious scripts probe for exploits. Scrapers download the same page dozens of times each second, or request the 10,000th page of comments for a post with only 2 comments.

Security Etudes : #101 : Security : Autoban

AutoBan

Self-learning intrusion detection in 100 lines of Perl

This is Part 2.

Basic design

• Set up rsyslog to route all http logs and all system logs (including auth) to a single /var/log/syslog file
• Create 2 text files: 'ignore.txt' and 'attack.txt'
• Have an always-on perl script 'aban.pl' that is looking at lines, that come into /var/log/syslog (realtime)
• In 'aban.pl', for every incoming line:
• ignore the line if the line matches one of ignore patterns
• store the line in 'attack.log' if the line matches one of attack patterns
• store the line in 'log' if it is neither junk, nor attack

Now we accumulate attacks in file attack.log, but how do we learn about attacks that we don't know?

We learn from the attackers!

• Introduce one more script - 'find-attacks.pl'
• In 'find-attacks.pl', for every line in 'attack.log'
• Extract ip address of attacker
• Search for that ip address in 'log' file (that contains lines that are not yet classified)
• With probability close to 100% all the requests from a criminal will also be criminal requests
• Look at the new criminal requests, isolate new patterns and add them to attack.txt
• Repeat

The Scripts

aban.pl

#!/usr/bin/perl

use File::Tail;

my $fname = shift || die "Use: $0 file to watch";

sub ReadCfg{
    my $fn = shift;
    my @ret = ();
    my @lines = `cat $fn`;
    foreach my $l (@lines) {
        chop $l;
        next if (!$l);
        next if ($l =~ m/^#.*/);
        push @ret, $l;
    }
    return @ret;
}

my @ignore=ReadCfg("ignore.txt");
my @attack=ReadCfg("attack.txt");

my $nrules = scalar @ignore;
my $narules = scalar @attack;
my $ldate = localtime;

open( LOG, ">>log" );
print LOG "$ldate started $0  watching '$fname' with $nrules ignore and $narules attack rules\n";
close( LOG );

my $File = File::Tail->new(name=>$fname, tail=>1, maxinterval=>0.5, interval=>0.5);
die "Can't read $fname" if (!$File);

while (defined($_=$File->read)) {
    chomp;
    my $line = $_;

    print STDERR "L: $line\n";

    foreach my $ire (@ignore){
        goto CONT if ( $line =~ m/\Q$ire\E/ );
    }

    foreach my $at (@attack){
        if ( $line =~ m/\Q$at\E/ ){
            open( ALOG, ">>attack.log" );
            print ALOG "$line\n";
            close( ALOG );
            goto CONT;
        }
    }

    open( LOG, ">>log" );
    print LOG "$line\n";
    close( LOG );

CONT:
}

find-attack.pl

#!/usr/bin/perl 

sub ReadCfg{
    my $fn = shift;
    my @ret = ();
    my @lines = `cat $fn`;
    foreach my $l (@lines) {
        chop $l;
        next if (!$l);
        next if ($l =~ m/^#.*/);
        push @ret, $l;
    }
    return @ret;
}

my @attack=ReadCfg("attack.txt");

my @lines = `cat attack.log`;
foreach my $l (@lines){
    chop $l;
    if ( $l =~  /(\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b)/ ){
        my $ip = $1;
        my $cmd = ( -f "old.log" ) ? "grep $ip log old.log" : "grep $ip log";
        my @morelines = `$cmd`;
        foreach my $ml (@morelines){
            chop $ml;
            foreach my $a (@attack){
                if ( $ml =~ m/\Q$a\E/ ){
                    # known attack
                    goto NEXTL;
                }
            }

            print "NEW ATTACK: $ml\n";

            NEXTL:
        }
    }
}

attack.txt

ic.asp
Invalid method in request
azenv.php
/forums
/config
HNAP1
anti-sec
admin
manager
pma
Failed password
user unknown
phppath
POST /%
noxdir
nosuichfile.php

July 28. 2013

Background

Summary Console - click to zoom in

Security Etudes : #102 : AutoBan Sneak Attacks

AutoBan Sneak Attacks

Self-learning intrusion detection in 100 lines of Perl

This is Part 3.

Some Attackers are Sneaky

• Most of attackers only hit you once from one IP
• Some attackers hit you from various IPs (rare, actually)
• Sneak attackers hit you several times from one IP, little by little, day by day
• You really want to block Sneak Attacker's IPs
• To detect Sneak Attackers, the summary.pl script is used
• Sample Summary Console is provided above

summary.pl

#!/usr/bin/perl 

use Data::Dumper;

my @lines=`cat /usr/aban/attack.log`;

my %i2n = ();
my %i2d = ();

foreach my $l (@lines) {
    chop $l;
    #print "$l\n";
    if ( $l =~  /(\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b)/ ){
        my $ip = $1;
        $i2n{ $ip }++;

        my $day = substr($l,0,6);
        #print "$day\n";
        if ( $i2d{$ip} && ($i2d{$ip} ne $day) ){
            $i2d{ $ip } .= " - " . $day;
        } else {
            $i2d{ $ip } = $day;
        }
    }

}


foreach my $i ( sort { $i2n{$b} <=> $i2n{$a} }  keys %i2n ){
    my $n = $i2n{$i};
    my $d = $i2d{$i};
    print "$i -> $n ($d)\n";
}

October 27. 2013

Background

Classification Console - click to zoom in

Security Etudes : #103 : AutoBan Layered Approach

AutoBan Layered Approach

So I blocked AWS (and China). You can too

This is Part 4.

Current AutoBan Layered Architecture

• Entire AWS (and entire China) are blocked via iptables. here is the script that does it.
• The system still learns from attackers (described in Part 1-3), but it only blocks annoying/sneak attackers.
• Attacker is "annoying" if it did some nasty in the past. Most of attacks are one time deals/bots.

I am looking for users to test client/server architecture

The system works very nice for me. No noise. Attakers are detected. Annoying attackers are blocked. Database of attacks signatures grows every week with no effort on my side.

I've been thinking about client/server architecture. In that scenario, users run a script on their computers, yet we share the database of attacks. If there is enough of users, it looks possible to detect attacking IPs and block them in realtime, the way yahoo and others fight spam. DM me on twitter (as of 2021 - no longer have twitter account), if you would want to secure your servers the way I secured mine. You don't need to be sysadmin for that, naturally.

I changed few things in 2021. Replaced the system with even more brute force approach, it bans aggressively. Automatically as well. Basically, a few strikes - IP is banned. Simpler that way. Works like a charm. (Aug 21 2021).

May 06. 2014

Background