HOWTO Setup fail2ban
What fail2ban does
fail2ban parses logfiles, and finds repeated-access-failures for various services. Once a specified number of failures (5) within a given time (10min) is reached, fail2ban makes an iptables-entry automatically, banning (blocking) that IP-address. After a configurable length of time (2 weeks), the IP-address is automatically unblocked. The status (number of hosts banned; number un-banned) is extracted from /var/log/fail2ban.log by logwatch, and summarized in our daily logwatch-email.
Why we want fail2ban
Our Gentoo systems use syslog-ng, sometimes with separate SSH auth.log file (many times, though SSH logging is integrated into /var/log/messages). Occasionally we also supply vsftpd connectivity for users, which uses yet another separate log-file. We may also add other services (apache2, postfix, and others). Fail2ban is capable of working with multiple log-files, and multiple services. The flexibility and integration with iptables/netfilter is a major benefit of fail2ban, and filtering is performed at the kernel-level. For most servers, this is the way to go.
Yet another alternative to fail2ban is a set of iptables rules, in which you specify a time-window and number-of-failures you will tolerate within that time-window. If you exercise some patience and wait a bit, the block goes away; most-effective against simplest automated attacks:
# define our own "extra" chain for processing SSH connections $IPTABLES -N SSHSCAN . . . $IPTABLES -A INPUT -p tcp --dport 22 -m state --state NEW -j SSHSCAN #IPTABLES -A INPUT -p tcp -m multiport --dports 22,2222 -m state --state NEW -j SSHSCAN $IPTABLES -A SSHSCAN -m recent --set --name SSH --rsource $IPTABLES -A SSHSCAN -m recent --update --seconds 10 --hitcount 4 --name SSH --rsource -j DROP #only 4 fails in 10s $IPTABLES -A SSHSCAN -m recent --update --seconds 1800 --hitcount 10 --name SSH --rsource -j DROP #only 10 fails in 30min $IPTABLES -A SSHSCAN -j ACCEPT #check status: cat /proc/self/net/xt_recent/SSH
Pre-Requisites for fail2ban
- Python
- iptables
- Note: if you want to use iptables-multiport (http, https for example) then verify that your kernel has Netfilter "multiport" included!)
- logrotate (watch out! Some stuff doesn't rotate; follow the link to fix it!)
- gamin (if you're following my suggestion, below)
Installing fail2ban
hostname ~ # emerge -v gamin fail2ban
Configuring fail2ban
previously, we edited /etc/fail2ban/jail.conf. But no more! Leave this jail.conf alone, so updates can adjust this, and our own local-config will persist.
Create a local-configuration file named jail.local:
hostname ~ # emacs -nw /etc/fail2ban/jail.local [DEFAULT] ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca bantime = 1209600 findtime = 1800 maxretry = 3 action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"] logpath = /var/log/auth.log logencoding=utf-8
[sshd] enabled = true [webmin-auth] enabled = true
Another example:
hostname ~ # emacs -nw /etc/fail2ban/jail.local [DEFAULT] ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca bantime = 1209600 findtime = 1800 maxretry = 3 action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"] # For hardened syslog-ng: #logpath = /var/log/auth.log # For non-hardened syslog-ng: logpath = /var/log/messages syslog_authpriv = /var/log/messages [sshd] enabled = true banaction = iptables-multiport port = 22,222
Running fail2ban
hostname ~ # /etc/init.d/fail2ban start * Starting fail2ban ... [ ok ]
hostname ~ # rc-update add fail2ban default * fail2ban added to runlevel default
Monitoring and Verifying fail2ban
Check the log file:
hostname ~ # tail /var/log/fail2ban.log
and, for scrolling/real-time monitoring of additions to the log-file:
hostname ~ # tail -f /var/log/fail2ban.log
Check Iptables:
hostname ~ # iptables -L
Example output, for a web-server (port 80 open), with SSH and FTP services too. Note the banned SSH user, resulting from too many failed-login-attempts:
Chain INPUT (policy DROP) target prot opt source destination fail2ban-VSFTPD tcp -- anywhere anywhere tcp dpt:ftp fail2ban-SSH tcp -- anywhere anywhere tcp dpt:ssh ACCEPT all -- anywhere anywhere DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ftp-data ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ftp ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED Chain fail2ban-SSH (1 references) target prot opt source destination DROP all -- sr-01504.iat.sfu.ca anywhere RETURN all -- anywhere anywhere Chain fail2ban-VSFTPD (1 references) target prot opt source destination RETURN all -- anywhere anywhere
Troubleshooting fail2ban
Log-file
By far, the most-common mistake is not specifying the proper log-file! Triple-check this, and use the next regex-tool to see that something sensible is happening.
Regular Expression Testing
One of the best ways of troubleshooting is to use the fail2ban-regex tool. Feed this with the log-file you want to watch, and point it to the filter you want applied:
hostname ~ # /usr/bin/fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/sshd.conf
Example output:
Unable to find a corresponding IP address for eges.esstel.ru
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Running tests
=============
Use regex file : /etc/fail2ban/filter.d/sshd.conf
Use log file : /var/log/messages
Results
=======
Failregex:
[1] Authentication failure for .* from <HOST>$
[2] Failed [-/\w]+ for .* from <HOST>$
[3] ROOT LOGIN REFUSED .* FROM <HOST>$
[4] [iI](?:llegal|nvalid) user .* from <HOST>$
Number of matches:
[1] 3161 match(es)
[2] 0 match(es)
[3] 0 match(es)
[4] 932 match(es)
Addresses found:
[1]
203.98.175.182 (Wed Nov 19 06:04:16 2008)
59.125.226.213 (Wed Nov 19 06:07:05 2008)
80.153.123.179 (Wed Nov 19 06:10:13 2008)
76.255.174.78 (Wed Nov 19 06:13:03 2008)
66.158.139.11 (Wed Nov 19 06:15:54 2008)
66.158.139.11 (Wed Nov 19 06:18:54 2008)
<truncated the rest of the output, for brevity's sake>
This verifies that the logfile you are parsing does indeed contain appropriate information for fail2ban, and it shows the filter-rules and numbers of matches for each rule. Some attacks try to sneak under a fail2ban threshold, so this kind of regex-test may show only a single, or perhaps 2 failures from a given IP-address... often (depending on configuration), this isn't enough to trigger a ban (the default config may require 5 attempts, or a tweaked version may reduce this threshold to 3 failures before banning).
Fail2ban Jail Status
Another useful tool is fail2ban-client to view the current status of a jail:
hostname ~ # /usr/bin/fail2ban-client status ssh-iptables
Example output:
Status for the jail: ssh-iptables |- filter | |- Currently failed: 11 | `- Total failed: 207 `- action |- Currently banned: 3 | `- IP list: 211.139.168.226 140.120.108.187 83.208.170.100 `- Total banned: 3
Other Things to Look For
If you see this kind of 0 references, and you're using iptables-multiport, check that you have enabled Netfilter "multiport" in your kernel:
... Chain fail2ban-ssh (0 references) target prot opt source destination RETURN all -- anywhere anywhere
0 references means that your actions are not in-effect :-O :-( You should not see a top-portion of your iptables that looks like this (without any fail2ban references):
hostname ~ # iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE DROP tcp -- anywhere anywhere tcp flags:FIN,SYN/FIN,SYN DROP tcp -- anywhere anywhere tcp flags:SYN,RST/SYN,RST DROP tcp -- anywhere anywhere tcp flags:FIN,RST/FIN,RST DROP tcp -- anywhere anywhere tcp flags:FIN,ACK/FIN DROP tcp -- anywhere anywhere tcp flags:PSH,ACK/PSH DROP tcp -- anywhere anywhere tcp flags:ACK,URG/URG ...
Instead, you should see an iptables top, and lower references which match up and look like this healthy example:
hostname ~ # iptables -L Chain INPUT (policy DROP) target prot opt source destination fail2ban-web tcp -- anywhere anywhere multiport dports http,https fail2ban-web tcp -- anywhere anywhere multiport dports http,https fail2ban-web tcp -- anywhere anywhere multiport dports http,https ... Chain fail2ban-web (3 references) target prot opt source destination RETURN all -- anywhere anywhere RETURN all -- anywhere anywhere RETURN all -- anywhere anywhere
Extra Fun!
You can use this one-liner to parse through your /var/log/messages file and totalize the number of times a specific IP-address has been attempting to access your machine:
awk '($(NF-7) = /invalid user/){print $(NF-3)}' /var/log/messages | sort | uniq -c | sort
Once you have this listing, you can manually add IP-blocks of the form:
iptables -I INPUT -p tcp -s 83.103.96.33 --dport ssh -j REJECT --reject-with tcp-reset