Veja lista de permissões no site do fail2ban:
# This will ignore connection coming from common private networks.
# Note that local connections can come from other than just 127.0.0.1, so
# this needs CIDR range too.
ignoreip = 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
Outra referência aqui :
First, find ignoreip. It's always important for you to have a way in! These are IPs are fail2ban will ignore - IPs listed here can always have invalid login attempts and still not be blocked. In my file, I'm putting down the network ranges for my internal network (192.168.1.0/24) as well as one other trusted IP address of a machine that I will be able to SSH into if need be. These need to be space separated! If they are not, fail2ban won't block anyone.