HOWTO Setup fail2ban: Difference between revisions

From Research
Jump to navigation Jump to search
← Older edit
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
==What fail2ban does==
==What fail2ban does==
'''''fail2ban''''' parses logfiles, and finds repeated-access-failures for various services.  Once a specified number of failures (5) within a given time (10min) is reached, fail2ban makes an iptables-entry automatically, banning (blocking) that IP-address.  After a configurable length of time (2 weeks), the IP-address is automatically unblocked.  The status (number of hosts banned; number un-banned) is extracted from /var/log/fail2ban by logwatch, and summarized in our daily logwatch-email.
'''''fail2ban''''' parses logfiles, and finds repeated-access-failures for various services.  Once a specified number of failures (5) within a given time (10min) is reached, fail2ban makes an iptables-entry automatically, banning (blocking) that IP-address.  After a configurable length of time (2 weeks), the IP-address is automatically unblocked.  The status (number of hosts banned; number un-banned) is extracted from /var/log/fail2ban.log by logwatch, and summarized in our daily logwatch-email.


==Why we want fail2ban==
==Why we want fail2ban==


Our Gentoo systems use syslog-ng, sometimes with separate SSH auth.log file (many times, though SSH logging is integrated into /var/log/messages).  Occasionally we also supply vsftpd connectivity for users, which uses yet another separate log-file.  We may also add other services (apache2, postfix, and others).  Fail2ban is capable of working with multiple log-files, and multiple services.  The flexibility and integration with iptables/netfilter is a major benefit of fail2ban, and filtering is performed at the kernel-level.  For most '''servers''', this is the way to go.<br><br>
Our Gentoo systems use syslog-ng, sometimes with separate SSH auth.log file (many times, though SSH logging is integrated into /var/log/messages).  Occasionally we also supply vsftpd connectivity for users, which uses yet another separate log-file.  We may also add other services (apache2, postfix, and others).  Fail2ban is capable of working with multiple log-files, and multiple services.  The flexibility and integration with iptables/netfilter is a major benefit of fail2ban, and filtering is performed at the kernel-level.  For most '''servers''', this is the way to go.<br>
'''''fail2ban''''' creates it's own logfile ('''/var/log/fail2ban.log'''), which is scanned and reported via '''''logwatch''''' every morning.  So, we get a previous-day summary, at no extra cost :-)<br>
 
<br>
<br>
<hr>
<hr>
One alternative to '''''fail2ban''''' is '''''denyhosts''''', which requires tcpwrappers ('''tcpd''' USE-flag), and makes banned-host-IP entries in your /etc/hosts.deny file.<br>
Yet another alternative to '''''fail2ban''''' is a set of iptables rules, in which you specify a time-window and number-of-failures you will tolerate within that time-window. If you exercise some patience and wait a bit, the block goes away; most-effective against simplest automated attacks:
*DenyHosts Advantages
 
**doesn't require iptables
# define our own "extra" chain for processing SSH connections
**scans your log-files for SSH failures only, and doesn't (appear) to care about the actual port.  This is very handy for SSH running on multiple, or non-standard ports.
$IPTABLES -N SSHSCAN
**can use an Internet server + database of abusive IP's, so it offers a bit more-proactive protection, rather than being purely reactionary
.
**it's just a Python script, and easily adapted to some of our legacy systems (SuSE 9.3 is a working example)
.
*DenyHosts Disadvantages
  .
**filtering is performed at the user-level.
$IPTABLES -A INPUT -p tcp --dport 22 -m state --state NEW -j SSHSCAN
**only capable of scanning a single log-file
#IPTABLES -A INPUT -p tcp -m multiport --dports 22,2222 -m state --state NEW -j SSHSCAN
**only supplied with SSH scanning support<br><br>
$IPTABLES -A SSHSCAN -m recent --set --name SSH --rsource
I would recommend DenyHosts for a '''desktop/workstation''', where SSH is the '''only''' externally-accessible service.  It is simpler, and informal testing has shown it works very well.<br>
$IPTABLES -A SSHSCAN -m recent --update --seconds 10 --hitcount 4 --name SSH --rsource -j DROP  #only 4 fails in 10s
$IPTABLES -A SSHSCAN -m recent --update --seconds 1800 --hitcount 10 --name SSH --rsource -j DROP  #only 10 fails in 30min
$IPTABLES -A SSHSCAN -j ACCEPT #check status:  cat /proc/self/net/xt_recent/SSH
 
<br>
<br>


==Pre-Requisites for fail2ban==
==Pre-Requisites for fail2ban==
*Python
*Python
*iptables
*[[HOWTO Setup iptables|iptables]]
**Note:  if you want to use iptables-multiport (http, https for example) then verify that your kernel has Netfilter "multiport" included!)
*[[HOWTO Setup Logrotate|logrotate]]  (watch out!  Some stuff doesn't rotate; follow the link to fix it!)
*[[HOWTO Setup Logrotate|logrotate]]  (watch out!  Some stuff doesn't rotate; follow the link to fix it!)
*gamin (if you're following my suggestion, below)
*gamin (if you're following my suggestion, below)


==Installing fail2ban==
==Installing fail2ban==
  <font color=red>hostname</font> <font color=blue>~ #</font> '''emerge -v fail2ban'''
  <font color=red>hostname</font> <font color=blue>~ #</font> '''emerge -v gamin fail2ban'''
<font color=red>hostname</font> <font color=blue>~ #</font> '''emacs -nw /etc/fail2ban/jail.conf'''


==Configuring fail2ban==
==Configuring fail2ban==
edit '''/etc/fail2ban/jail.conf'''
previously, we edited '''/etc/fail2ban/jail.conf'''.  But no more!  Leave this jail.conf alone, so updates can adjust this, and our own local-config will persist.
*change how we monitor failures, preferring a file-alteration monitor over polling (you may have to '''emerge gamin'''):
###backend = auto
backend = gamin


*scroll to section "[ssh-iptables]" and enable it:
Create a local-configuration file named jail.local:
###enabled  = false
enabled  = true


*change the the SSH logfile we want to scan, if necessary (often it is not; sometimes it is):
  <font color=red>hostname</font> <font color=blue>~ #</font> '''emacs -nw /etc/fail2ban/jail.local'''
  ###logpath = /var/log/sshd.log
[DEFAULT]
  logpath = /var/log/auth.log
ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca
bantime  = 1209600
findtime  = 1800
maxretry = 3
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
  logpath = /var/log/auth.log
logencoding=utf-8


*comment out the "mail-whois" actions, in order to reduce email-clutter. If you do wish to be notified each time an IP-address is blocked, edit jail.conf and replace ''you@mail.com'' with your own email-address.<br>
[sshd]
enabled  = true
[webmin-auth]
  enabled = true


Repeat the enabling and configuring for all the other services you want, such as vsftpd, apache, etc.<br>
Another example:
<font color=red>hostname</font> <font color=blue>~ #</font> '''emacs -nw /etc/fail2ban/jail.local'''
[DEFAULT]
ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca
bantime  = 1209600
findtime  = 1800
maxretry = 3
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
# For hardened syslog-ng:
#logpath  = /var/log/auth.log
# For non-hardened syslog-ng:
logpath  = /var/log/messages
syslog_authpriv = /var/log/messages
[sshd]
enabled  = true
banaction = iptables-multiport
port   = 22,222


==Running fail2ban==
==Running fail2ban==
Line 93: Line 123:
  target    prot opt source              destination         
  target    prot opt source              destination         
  RETURN    all  --  anywhere            anywhere
  RETURN    all  --  anywhere            anywhere
==Troubleshooting fail2ban==
===Log-file===
By far, the most-common mistake is not specifying the proper log-file!  Triple-check this, and use the next regex-tool to see that something sensible is happening.
===Regular Expression Testing===
One of the best ways of troubleshooting is to use the '''fail2ban-regex tool'''.  Feed this with the log-file you want to watch, and point it to the filter you want applied:
<font color=red>hostname</font> <font color=blue>~ #</font> '''/usr/bin/fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/sshd.conf'''
Example output:
Unable to find a corresponding IP address for eges.esstel.ru
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Running tests
=============
Use regex file : /etc/fail2ban/filter.d/sshd.conf
Use log file  : /var/log/messages
Results
=======
Failregex:
[1] Authentication failure for .* from <HOST>$
[2] Failed [-/\w]+ for .* from <HOST>$
[3] ROOT LOGIN REFUSED .* FROM <HOST>$
[4] [iI](?:llegal|nvalid) user .* from <HOST>$
Number of matches:
[1] 3161 match(es)
[2] 0 match(es)
[3] 0 match(es)
[4] 932 match(es)
Addresses found:
[1]
    203.98.175.182 (Wed Nov 19 06:04:16 2008)
    59.125.226.213 (Wed Nov 19 06:07:05 2008)
    80.153.123.179 (Wed Nov 19 06:10:13 2008)
    76.255.174.78 (Wed Nov 19 06:13:03 2008)
    66.158.139.11 (Wed Nov 19 06:15:54 2008)
    66.158.139.11 (Wed Nov 19 06:18:54 2008)
<truncated the rest of the output, for brevity's sake>
This verifies that the logfile you are parsing does indeed contain appropriate information for fail2ban, and it shows the filter-rules and numbers of matches for each rule.  Some attacks try to sneak under a fail2ban threshold, so this kind of regex-test may show only a single, or perhaps 2 failures from a given IP-address... often (depending on configuration), this isn't enough to trigger a ban (the default config may require 5 attempts, or a tweaked version may reduce this threshold to 3 failures before banning).
<br>
<br>
===Fail2ban Jail Status===
Another useful tool is '''fail2ban-client''' to view the current status of a jail:
<font color=red>hostname</font> <font color=blue>~ #</font> '''/usr/bin/fail2ban-client status ssh-iptables'''
Example output:
Status for the jail: ssh-iptables
|- filter
|  |- Currently failed: 11
|  `- Total failed: 207
`- action
    |- Currently banned: 3
    |  `- IP list: 211.139.168.226 140.120.108.187 83.208.170.100
    `- Total banned: 3
<br>
===Other Things to Look For===
If you see this kind of 0 references, and you're using iptables-multiport, check that you have enabled Netfilter "multiport" in your kernel:
...
Chain fail2ban-ssh (0 references)
target    prot opt source              destination       
RETURN    all  --  anywhere            anywhere
'''0 references''' means that your actions are '''not''' in-effect :-O :-(  You '''should not''' see a top-portion of your iptables that looks like this (without any fail2ban references):
<font color=red>hostname</font> <font color=blue>~ #</font> '''iptables -L'''
Chain INPUT (policy DROP)
target    prot opt source              destination       
ACCEPT    all  --  anywhere            anywhere           
DROP      tcp  --  anywhere            anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP      tcp  --  anywhere            anywhere            tcp flags:FIN,SYN/FIN,SYN
DROP      tcp  --  anywhere            anywhere            tcp flags:SYN,RST/SYN,RST
DROP      tcp  --  anywhere            anywhere            tcp flags:FIN,RST/FIN,RST
DROP      tcp  --  anywhere            anywhere            tcp flags:FIN,ACK/FIN
DROP      tcp  --  anywhere            anywhere            tcp flags:PSH,ACK/PSH
DROP      tcp  --  anywhere            anywhere            tcp flags:ACK,URG/URG
...
Instead, you should see an iptables top, and lower references which match up and look like this healthy example:
<font color=red>hostname</font> <font color=blue>~ #</font> '''iptables -L'''
Chain INPUT (policy DROP)
target    prot opt source              destination       
fail2ban-web  tcp  --  anywhere            anywhere            multiport dports http,https
fail2ban-web  tcp  --  anywhere            anywhere            multiport dports http,https
fail2ban-web  tcp  --  anywhere            anywhere            multiport dports http,https
...
Chain fail2ban-web (3 references)
target    prot opt source              destination       
RETURN    all  --  anywhere            anywhere           
RETURN    all  --  anywhere            anywhere           
RETURN    all  --  anywhere            anywhere
==Extra Fun!==
You can use this one-liner to parse through your '''/var/log/messages''' file and totalize the number of times a specific IP-address has been attempting to access your machine:
awk '($(NF-7) = /invalid user/){print $(NF-3)}' /var/log/messages | sort | uniq -c | sort
Once you have this listing, you can manually add IP-blocks of the form:
iptables -I INPUT -p tcp -s 83.103.96.33 --dport ssh -j REJECT --reject-with tcp-reset

Latest revision as of 23:37, 27 February 2017

What fail2ban does

fail2ban parses logfiles, and finds repeated-access-failures for various services. Once a specified number of failures (5) within a given time (10min) is reached, fail2ban makes an iptables-entry automatically, banning (blocking) that IP-address. After a configurable length of time (2 weeks), the IP-address is automatically unblocked. The status (number of hosts banned; number un-banned) is extracted from /var/log/fail2ban.log by logwatch, and summarized in our daily logwatch-email.

Why we want fail2ban

Our Gentoo systems use syslog-ng, sometimes with separate SSH auth.log file (many times, though SSH logging is integrated into /var/log/messages). Occasionally we also supply vsftpd connectivity for users, which uses yet another separate log-file. We may also add other services (apache2, postfix, and others). Fail2ban is capable of working with multiple log-files, and multiple services. The flexibility and integration with iptables/netfilter is a major benefit of fail2ban, and filtering is performed at the kernel-level. For most servers, this is the way to go.


Yet another alternative to fail2ban is a set of iptables rules, in which you specify a time-window and number-of-failures you will tolerate within that time-window. If you exercise some patience and wait a bit, the block goes away; most-effective against simplest automated attacks: 

# define our own "extra" chain for processing SSH connections
$IPTABLES -N SSHSCAN
.
.
.
$IPTABLES -A INPUT -p tcp --dport 22 -m state --state NEW -j SSHSCAN
#IPTABLES -A INPUT -p tcp -m multiport --dports 22,2222 -m state --state NEW -j SSHSCAN
$IPTABLES -A SSHSCAN -m recent --set --name SSH --rsource
$IPTABLES -A SSHSCAN -m recent --update --seconds 10 --hitcount 4 --name SSH --rsource -j DROP  #only 4 fails in 10s
$IPTABLES -A SSHSCAN -m recent --update --seconds 1800 --hitcount 10 --name SSH --rsource -j DROP  #only 10 fails in 30min
$IPTABLES -A SSHSCAN -j ACCEPT #check status:  cat /proc/self/net/xt_recent/SSH



Pre-Requisites for fail2ban

  • Python
  • iptables
    • Note: if you want to use iptables-multiport (http, https for example) then verify that your kernel has Netfilter "multiport" included!)
  • logrotate (watch out! Some stuff doesn't rotate; follow the link to fix it!)
  • gamin (if you're following my suggestion, below)

Installing fail2ban

hostname ~ # emerge -v gamin fail2ban

Configuring fail2ban

previously, we edited /etc/fail2ban/jail.conf. But no more! Leave this jail.conf alone, so updates can adjust this, and our own local-config will persist.

Create a local-configuration file named jail.local: 

hostname ~ # emacs -nw /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca
bantime  = 1209600
findtime  = 1800
maxretry = 3
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
logpath  = /var/log/auth.log
logencoding=utf-8

[sshd]
enabled  = true

[webmin-auth]
enabled = true

Another example: 

hostname ~ # emacs -nw /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = sfu.ca shaw.ca telus.ca telus.com teksavvy.com rogers.com rogers.ca bell.ca
bantime  = 1209600
findtime  = 1800
maxretry = 3
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

# For hardened syslog-ng:
#logpath  = /var/log/auth.log

# For non-hardened syslog-ng:
logpath  = /var/log/messages
syslog_authpriv = /var/log/messages

[sshd]
enabled  = true
banaction = iptables-multiport
port	   = 22,222

Running fail2ban

hostname ~ # /etc/init.d/fail2ban start
* Starting fail2ban ...                                         [ ok ]

hostname ~ # rc-update add fail2ban default
* fail2ban added to runlevel default

Monitoring and Verifying fail2ban

Check the log file: 

hostname ~ # tail /var/log/fail2ban.log

and, for scrolling/real-time monitoring of additions to the log-file: 

hostname ~ # tail -f /var/log/fail2ban.log

Check Iptables: 

hostname ~ # iptables -L

Example output, for a web-server (port 80 open), with SSH and FTP services too. Note the banned SSH user, resulting from too many failed-login-attempts: 

Chain INPUT (policy DROP)
target     prot opt source               destination         
fail2ban-VSFTPD  tcp  --  anywhere             anywhere            tcp dpt:ftp 
fail2ban-SSH  tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            state INVALID 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp-data 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 

Chain fail2ban-SSH (1 references)
target     prot opt source               destination         
DROP       all  --  sr-01504.iat.sfu.ca  anywhere            
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-VSFTPD (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Troubleshooting fail2ban

Log-file

By far, the most-common mistake is not specifying the proper log-file! Triple-check this, and use the next regex-tool to see that something sensible is happening.

Regular Expression Testing

One of the best ways of troubleshooting is to use the fail2ban-regex tool. Feed this with the log-file you want to watch, and point it to the filter you want applied: 

hostname ~ # /usr/bin/fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/sshd.conf

Example output: 

Unable to find a corresponding IP address for eges.esstel.ru
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net
Unable to find a corresponding IP address for adsl-75-22-172-193.dsl.sndg02.sbcglobal.net

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/sshd.conf
Use log file   : /var/log/messages


Results
=======

Failregex:
[1] Authentication failure for .* from <HOST>$
[2] Failed [-/\w]+ for .* from <HOST>$
[3] ROOT LOGIN REFUSED .* FROM <HOST>$
[4] [iI](?:llegal|nvalid) user .* from <HOST>$

Number of matches:
[1] 3161 match(es)
[2] 0 match(es)
[3] 0 match(es)
[4] 932 match(es)

Addresses found:
[1]
    203.98.175.182 (Wed Nov 19 06:04:16 2008)
    59.125.226.213 (Wed Nov 19 06:07:05 2008)
    80.153.123.179 (Wed Nov 19 06:10:13 2008)
    76.255.174.78 (Wed Nov 19 06:13:03 2008)
    66.158.139.11 (Wed Nov 19 06:15:54 2008)
    66.158.139.11 (Wed Nov 19 06:18:54 2008)

<truncated the rest of the output, for brevity's sake>

This verifies that the logfile you are parsing does indeed contain appropriate information for fail2ban, and it shows the filter-rules and numbers of matches for each rule. Some attacks try to sneak under a fail2ban threshold, so this kind of regex-test may show only a single, or perhaps 2 failures from a given IP-address... often (depending on configuration), this isn't enough to trigger a ban (the default config may require 5 attempts, or a tweaked version may reduce this threshold to 3 failures before banning).

Fail2ban Jail Status

Another useful tool is fail2ban-client to view the current status of a jail: 

hostname ~ # /usr/bin/fail2ban-client status ssh-iptables

Example output: 

Status for the jail: ssh-iptables
|- filter
|  |- Currently failed:	11
|  `- Total failed:	207
`- action
   |- Currently banned:	3
   |  `- IP list:	211.139.168.226 140.120.108.187 83.208.170.100 
   `- Total banned:	3


Other Things to Look For

If you see this kind of 0 references, and you're using iptables-multiport, check that you have enabled Netfilter "multiport" in your kernel: 

...
Chain fail2ban-ssh (0 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

0 references means that your actions are not in-effect :-O :-( You should not see a top-portion of your iptables that looks like this (without any fail2ban references): 

hostname ~ # iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN/FIN,SYN 
DROP       tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN,RST 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,RST/FIN,RST 
DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,ACK/FIN 
DROP       tcp  --  anywhere             anywhere            tcp flags:PSH,ACK/PSH 
DROP       tcp  --  anywhere             anywhere            tcp flags:ACK,URG/URG 
...

Instead, you should see an iptables top, and lower references which match up and look like this healthy example: 

hostname ~ # iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
fail2ban-web  tcp  --  anywhere             anywhere            multiport dports http,https 
...
Chain fail2ban-web (3 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere

Extra Fun!

You can use this one-liner to parse through your /var/log/messages file and totalize the number of times a specific IP-address has been attempting to access your machine: 

awk '($(NF-7) = /invalid user/){print $(NF-3)}' /var/log/messages | sort | uniq -c | sort

Once you have this listing, you can manually add IP-blocks of the form: 

iptables -I INPUT -p tcp -s 83.103.96.33 --dport ssh -j REJECT --reject-with tcp-reset
Retrieved from "https://research.iat.sfu.ca/research/index.php?title=HOWTO_Setup_fail2ban&oldid=5840"