HOWTO Setup fail2ban

What fail2ban does

fail2ban parses logfiles, and finds repeated-access-failures for various services. Once a specified number of failures (5) within a given time (10min) is reached, fail2ban makes an iptables-entry automatically, banning (blocking) that IP-address. After a configurable length of time (2 weeks), the IP-address is automatically unblocked. The status (number of hosts banned; number un-banned) is extracted from /var/log/fail2ban.log by logwatch, and summarized in our daily logwatch-email.

Why we want fail2ban

Our Gentoo systems use syslog-ng, sometimes with separate SSH auth.log file (many times, though SSH logging is integrated into /var/log/messages). Occasionally we also supply vsftpd connectivity for users, which uses yet another separate log-file. We may also add other services (apache2, postfix, and others). Fail2ban is capable of working with multiple log-files, and multiple services. The flexibility and integration with iptables/netfilter is a major benefit of fail2ban, and filtering is performed at the kernel-level. For most servers, this is the way to go.

---

One alternative to fail2ban is denyhosts, which requires tcpwrappers (tcpd USE-flag), and makes banned-host-IP entries in your /etc/hosts.deny file. I would recommend DenyHosts for a desktop/workstation, where SSH is the only externally-accessible service. It is simpler than fail2ban, and informal testing has shown it works very well.

• DenyHosts Advantages
  • doesn't require iptables
  • scans your log-files for SSH failures only, and doesn't (appear) to care about the actual port. This is very handy for SSH running on multiple, or non-standard ports.
  • can use an Internet server + database of abusive IP's, so it offers a bit more-proactive protection, rather than being purely reactionary
  • it's just a Python script, and easily adapted to some of our legacy systems (SuSE 9.3 is a working example)
• DenyHosts Disadvantages
  • filtering is performed at the user-level.
  • only capable of scanning a single log-file
  • only supplied with SSH scanning support
    

---

Yet another alternative to fail2ban is a simple iptable-rule, in which you specify a time-window and number-of-failures you will tolerate within that time-window. If you exercise some patience and wait a bit, the block goes away; most-effective against simplest automated attacks:

$IPTABLES -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
$IPTABLES -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --rttl --name SSH -j DROP 

Pre-Requisites for fail2ban

• Python
• iptables
  • Note: if you want to use iptables-multiport (http, https for example) then verify that your kernel has Netfilter "multiport" included!)
• logrotate (watch out! Some stuff doesn't rotate; follow the link to fix it!)
• gamin (if you're following my suggestion, below)

Installing fail2ban

hostname ~ # emerge -v gamin fail2ban

Configuring fail2ban

edit /etc/fail2ban/jail.conf

hostname ~ # emacs -nw /etc/fail2ban/jail.conf

• set the bantime to 2 weeks, measured in seconds:

bantime  = 1209600

• set the findtime to 30 minutes, again measured in seconds:

findtime  = 1800

• scroll to section "[ssh-iptables]" and enable it:

###enabled  = false
enabled  = true

• change the the SSH logfile we want to scan, if necessary (often it is not; sometimes it is):

###logpath = /var/log/sshd.log
logpath = /var/log/auth.log

• tighten up the max failures we'll tolerate; remember, on-campus and a select few others are ignored and not subject to this (allows fumble-fingered admins to successfully auth)

maxretry = 3

• comment out the "mail-whois" actions, in order to reduce email-clutter. If you do wish to be notified each time an IP-address is blocked, edit jail.conf and replace you@mail.com with your own email-address.
  

#           mail-whois[name=SSH, dest=yourmail@mail.com] 

Repeat the enabling and configuring for all the other services you want, such as vsftpd, apache, etc.

Extend the whitelist of IP-addresses we don't want to block. BEWARE - after the default ignoreip definition, any later occurrence will override and obliterate the first ignoreip definition. This can be handy for a per-service whitelist, but you cannot span your whitelist across multiple lines to enhance readability... Use just one super-long ugly entry, and use comments instead for your readability needs:

# SFU IP ADDRESSES.  Taken from http://whois.arin.net/rest/org/SFU-1/nets on March 6, 2012. GRP.
#ignoreip = 192.75.241.0/11 142.58.0.0/16  209.87.56.0/24 206.12.128.0/8 206.12.30.0/8 207.23.85.0/8 204.244.192.168/3 199.60.0.0/12

# SHAW IP ADDRESSES.  Taken from http://whois.arin.net/rest/org/SHAWC/nets on March 6, 2012. GRP.
#ignoreip = 174.0.0.0/13 184.64.0.0/13 204.209.208.0/21 204.244.240.0/9 24.108.0.0/15 24.64.0.0/13 24.70.0.0/15 24.71.223.0/24 24.76.0.0/15 24.80.0.0/13 24.244.0.0/18 50.64.0.0/13 68.144.0.0/11 70.64.0.0/12 96.48.0.0/13 64.59.128.0/18 66.163.64.0/20 

# TELUS IP ADDRESSES.  Taken from http://whois.arin.net/rest/org/TACE/nets on March 6, 2012. GRP.
#ignoreip = 204.209.208.0/11 24.108.0.0/15 64.59.128.0/18 142.60.0.0/16 137.186.0.0/16 206.75.0.0/16 207.81.0.0/16 207.134.0.0/16 154.5.0.0/16 204.174.120.0/15 209.115.222.0/8 204.174.64.0/11 142.169.0.0/16 142.168.0.0/16 199.84.240.0/10 64.180.0.0/16 50.92.0.0/15 216.226.32.0/13 206.162.128.0/14 66.110.128.0/14 207.102.0.0/16 206.108.16.0/12 207.34.128.0/14 207.194.0.0/16 208.181.0.0/16 209.52.0.0/16 204.174.211.0/11 206.108.192.0/13 64.114.0.0/16 209.115.128.0/15 161.184.64.0/11 209.107.96.0/13 154.5.0.0/16 207.228.64.0/14 207.148.128.0/14 209.91.64.0/14 205.236.24.0/8 205.236.48.0/8 209.104.64.0/14 207.167.192.0/14 154.11.0.0/16 216.123.192.0/14 207.229.0.0/14 208.38.0.0/14 199.212.152.0/11 204.225.240.0/11 207.34.192.0/14 209.20.0.0/14 209.121.0.0/16 206.108.64.0/13 209.89.0.0/16 205.206.0.0/16 209.162.160.0/12 204.191.0.0/16 205.250.0.0/16 207.81.0.0/16 209.171.0.0/16 209.29.0.0/16 198.53.0.0/16 206.116.0.0/16 198.166.0.0/16 206.75.0.0/16 207.216.0.0/16 207.6.0.0/16 66.222.128.0/15 207.134.0.0/16 216.218.0.0/14 137.186.0.0/16 199.126.0.0/16 209.202.64.0/14 66.203.192.0/13 207.219.0.0/16 216.232.0.0/16 209.53.0.0/16 66.183.0.0/16 207.34.64.0/14 206.26.204.192/14 205.138.102.0/8 207.6.32.0/13 206.116.224.0/11 173.180.0.0/16 205.138.98.0/8 108.172.0.0/16 75.152.0.0/16 154.20.0.0/16 99.199.0.0/16 50.98.0.0/16 23.16.0.0/16

# DELTA CABLE IP ADDRESSES.  Taken from http://whois.arin.net/rest/org/DLTA/nets on March 6, 2012. GRP.
#ignoreip = 24.207.0.0/15

ignoreip = 127.0.0.1/8 192.75.241.0/11 142.58.0.0/16  209.87.56.0/24 206.12.128.0/8 206.12.30.0/8 207.23.85.0/8 204.244.192.168/3 199.60.0.0/12 174.0.0.0/13 184.64.0.0/13 204.209.208.0/21 204.244.240.0/9 24.108.0.0/15 24.64.0.0/13 24.70.0.0/15 24.71.223.0/24 24.76.0.0/15 24.80.0.0/13 24.244.0.0/18 50.64.0.0/13 68.144.0.0/11 70.64.0.0/12 96.48.0.0/13 64.59.128.0/18 66.163.64.0/20 204.209.208.0/11 24.108.0.0/15 64.59.128.0/18 142.60.0.0/16 137.186.0.0/16 206.75.0.0/16 207.81.0.0/16 207.134.0.0/16 154.5.0.0/16 204.174.120.0/15 209.115.222.0/8 204.174.64.0/11 142.169.0.0/16 142.168.0.0/16 199.84.240.0/10 64.180.0.0/16 50.92.0.0/15 216.226.32.0/13 206.162.128.0/14 66.110.128.0/14 207.102.0.0/16 206.108.16.0/12 207.34.128.0/14 207.194.0.0/16 208.181.0.0/16 209.52.0.0/16 204.174.211.0/11 206.108.192.0/13 64.114.0.0/16 209.115.128.0/15 161.184.64.0/11 209.107.96.0/13 154.5.0.0/16 207.228.64.0/14 207.148.128.0/14 209.91.64.0/14 205.236.24.0/8 205.236.48.0/8 209.104.64.0/14 207.167.192.0/14 154.11.0.0/16 216.123.192.0/14 207.229.0.0/14 208.38.0.0/14 199.212.152.0/11 204.225.240.0/11 207.34.192.0/14 209.20.0.0/14 209.121.0.0/16 206.108.64.0/13 209.89.0.0/16 205.206.0.0/16 209.162.160.0/12 204.191.0.0/16 205.250.0.0/16 207.81.0.0/16 209.171.0.0/16 209.29.0.0/16 198.53.0.0/16 206.116.0.0/16 198.166.0.0/16 206.75.0.0/16 207.216.0.0/16 207.6.0.0/16 66.222.128.0/15 207.134.0.0/16 216.218.0.0/14 137.186.0.0/16 199.126.0.0/16 209.202.64.0/14 66.203.192.0/13 207.219.0.0/16 216.232.0.0/16 209.53.0.0/16 66.183.0.0/16 207.34.64.0/14 206.26.204.192/14 205.138.102.0/8 207.6.32.0/13 206.116.224.0/11 173.180.0.0/16 205.138.98.0/8 108.172.0.0/16 75.152.0.0/16 154.20.0.0/16 99.199.0.0/16 50.98.0.0/16 23.16.0.0/16 24.207.0.0/15

Running fail2ban

hostname ~ # /etc/init.d/fail2ban start
* Starting fail2ban ...                                         [ ok ]

hostname ~ # rc-update add fail2ban default
* fail2ban added to runlevel default

Monitoring and Verifying fail2ban

Check the log file:

hostname ~ # tail /var/log/fail2ban.log

and, for scrolling/real-time monitoring of additions to the log-file:

hostname ~ # tail -f /var/log/fail2ban.log

Check Iptables:

hostname ~ # iptables -L

Example output, for a web-server (port 80 open), with SSH and FTP services too. Note the banned SSH user, resulting from too many failed-login-attempts:

Chain INPUT (policy DROP)
target     prot opt source               destination         
fail2ban-VSFTPD  tcp  --  anywhere             anywhere            tcp dpt:ftp 
fail2ban-SSH  tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            state INVALID 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp-data 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 

Chain fail2ban-SSH (1 references)
target     prot opt source               destination         
DROP       all  --  sr-01504.iat.sfu.ca  anywhere            
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-VSFTPD (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Troubleshooting fail2ban

Log-file

By far, the most-common mistake is not specifying the proper log-file! Triple-check this, and use the next regex-tool to see that something sensible is happening.

Regular Expression Testing

One of the best ways of troubleshooting is to use the fail2ban-regex tool. Feed this with the log-file you want to watch, and point it to the filter you want applied:

hostname ~ # /usr/bin/fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/sshd.conf

Example output:

Unable to find a corresponding IP address for eges.esstel.ru
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/sshd.conf
Use log file   : /var/log/messages


Results
=======

Failregex:
[1] Authentication failure for .* from <HOST>$
[2] Failed [-/\w]+ for .* from <HOST>$
[3] ROOT LOGIN REFUSED .* FROM <HOST>$
[4] [iI](?:llegal|nvalid) user .* from <HOST>$

Number of matches:
[1] 3161 match(es)
[2] 0 match(es)
[3] 0 match(es)
[4] 932 match(es)

Addresses found:
[1]
    203.98.175.182 (Wed Nov 19 06:04:16 2008)
    59.125.226.213 (Wed Nov 19 06:07:05 2008)
    80.153.123.179 (Wed Nov 19 06:10:13 2008)
    76.255.174.78 (Wed Nov 19 06:13:03 2008)
    66.158.139.11 (Wed Nov 19 06:15:54 2008)
    66.158.139.11 (Wed Nov 19 06:18:54 2008)

<truncated the rest of the output, for brevity's sake>

This verifies that the logfile you are parsing does indeed contain appropriate information for fail2ban, and it shows the filter-rules and numbers of matches for each rule. Some attacks try to sneak under a fail2ban threshold, so this kind of regex-test may show only a single, or perhaps 2 failures from a given IP-address... often (depending on configuration), this isn't enough to trigger a ban (the default config may require 5 attempts, or a tweaked version may reduce this threshold to 3 failures before banning).

Fail2ban Jail Status

Another useful tool is fail2ban-client to view the current status of a jail:

hostname ~ # /usr/bin/fail2ban-client status ssh-iptables

Example output:

Status for the jail: ssh-iptables
|- filter
|  |- Currently failed:	11
|  `- Total failed:	207
`- action
   |- Currently banned:	3
   |  `- IP list:	211.139.168.226 140.120.108.187 83.208.170.100 
   `- Total banned:	3

Other Things to Look For

If you see this kind of 0 references, and you're using iptables-multiport, check that you have enabled Netfilter "multiport" in your kernel:

...
Chain fail2ban-ssh (0 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

0 references means that your actions are not in-effect :-O :-( You should not see a top-portion of your iptables that looks like this (without any fail2ban references):

hostname ~ # iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN/FIN,SYN 
DROP       tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN,RST 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,RST/FIN,RST 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,ACK/FIN 
DROP       tcp  --  anywhere             anywhere            tcp flags:PSH,ACK/PSH 
DROP       tcp  --  anywhere             anywhere            tcp flags:ACK,URG/URG 
...

Instead, you should see an iptables top, and lower references which match up and look like this healthy example:

hostname ~ # iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
...
Chain fail2ban-web (3 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere

Extra Fun!

You can use this one-liner to parse through your /var/log/messages file and totalize the number of times a specific IP-address has been attempting to access your machine:

awk '($(NF-7) = /invalid user/){print $(NF-3)}' /var/log/messages | sort | uniq -c | sort

Once you have this listing, you can manually add IP-blocks of the form:

iptables -I INPUT -p tcp -s 83.103.96.33 --dport ssh -j REJECT --reject-with tcp-reset