I had installed fail2ban but had noticed it wasn’t working blocking ssh brute force attacks. Attacks such as below.

grep sshd /var/log/auth.log | tail
Apr 29 08:06:17 sd-229337 sshd[20646]: pam_unix(sshd:auth): check pass; user unknown
Apr 29 08:06:17 sd-229337 sshd[20646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=211-75-3-35.hinet-ip.hinet.net
Apr 29 08:06:18 sd-229337 sshd[20646]: Failed password for invalid user db2inst from 211.75.3.35 port 52724 ssh2
Apr 29 08:06:19 sd-229337 sshd[20646]: Connection closed by 211.75.3.35 [preauth]
Apr 29 08:18:21 sd-229337 sshd[20711]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=59-120-243-8.hinet-ip.hinet.net  user=root
Apr 29 08:18:25 sd-229337 sshd[20711]: Failed password for root from 59.120.243.8 port 34312 ssh2
Apr 29 08:18:25 sd-229337 sshd[20711]: Connection closed by 59.120.243.8 [preauth]
Apr 29 08:19:14 sd-229337 sshd[20713]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=195-154-136-62.rev.poneytelecom.eu  user=root
Apr 29 08:19:16 sd-229337 sshd[20713]: Failed password for root from 195.154.136.62 port 24329 ssh2
Apr 29 08:19:16 sd-229337 sshd[20713]: Connection closed by 195.154.136.62 [preauth]

In order to fix this we need to modify /etc/fail2ban/filter.d/common.local and modify bsd_syslog_verbose entry. Substitute __bsd_syslog_verbose = (<[^.]+\.[^.]+>) for __bsd_syslog_verbose = (<[^.]+ [^.]+>).

grep bsd_syslog_verbose /etc/fail2ban/filter.d/common.local
#__bsd_syslog_verbose = (<[^.]+\.[^.]+>)
__bsd_syslog_verbose = (<[^.]+ [^.]+>)
__prefix_line = \s*%(__bsd_syslog_verbose)s?\s*(?:%(__hostname)s )?(?:%(__kernel_prefix)s )?(?:@vserver_\S+ )?%(__daemon_combs_re)s?\s%(__daemon_extra_re)s?\s*

Restart fail2ban and you shall now see IPs performing brute force attacks being blocked as below.

tail -30 /var/log/fail2ban.log | grep actions
2018-04-29 18:43:19,835 fail2ban.actions[28271]: WARNING [ssh] Unban 163.172.159.119
2018-04-29 18:43:20,742 fail2ban.actions[28519]: INFO    Set banTime = 1800
2018-04-29 18:43:20,936 fail2ban.actions[28519]: INFO    Set banTime = 600
2018-04-29 18:43:59,119 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195
2018-04-29 18:46:05,286 fail2ban.actions[28519]: WARNING [ssh] Ban 5.188.10.185
2018-04-29 19:13:59,938 fail2ban.actions[28519]: WARNING [ssh] Unban 171.244.27.195
2018-04-29 19:14:50,026 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195
2018-04-29 19:15:35,102 fail2ban.actions[28519]: WARNING [ssh] Ban 159.65.10.166
2018-04-29 19:16:06,167 fail2ban.actions[28519]: WARNING [ssh] Unban 5.188.10.185
2018-04-29 19:44:50,740 fail2ban.actions[28519]: WARNING [ssh] Unban 171.244.27.195
2018-04-29 19:45:35,821 fail2ban.actions[28519]: WARNING [ssh] Unban 159.65.10.166
2018-04-29 19:45:38,858 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195

But why is this happening? It is because of regular expressions. The way logs are being written it will never find a match with the original __bsd_syslog_verbose. Below script test both bsd_syslog_verbose settings. Originally we needed to have a ., but in reality we have a space in our logs, so we need to modify bsd_syslog_verbose.

#!/usr/bin/env python

import re

testline = 'May 13 06:24:36'

match = re.search('[^.]+\.[^.]+', testline)
if match:
    print 'Found:', match.group()
else:
    print 'Not found for bsd_syslog_verbose=[^.]+\.[^.]+'

match = re.search('[^.]+ [^.]+', testline)
if match:
    print 'Found:', match.group()
else:
    print 'Not found for bsd_syslog_verbose=[^.]+ [^.]+'

And we execute:

 python regex.py 
Not found for bsd_syslog_verbose=[^.]+\.[^.]+
Found: May 13 06:24:36

More info here and some instructive regex google doc.