você pode resolver este problema de uma maneira simples, tente usar o módulo recente com o iptables, recentemente rastreie o endereço de origem:
iptables -m recent -h
recent match options:
[!] --set Add source address to list, always matches.
[!] --rcheck Match if source address in list.
[!] --update Match if source address in list, also update last-seen time.
[!] --remove Match if source address in list, also removes that address from list.
--seconds seconds For check and update commands above.
Specifies that the match will only occur if source address last seen within
the last 'seconds' seconds.
--reap Purge entries older then 'seconds'.
Can only be used in conjunction with the seconds option.
--hitcount hits For check and update commands above.
Specifies that the match will only occur if source address seen hits times.
May be used in conjunction with the seconds option.
--rttl For check and update commands above.
Specifies that the match will only occur if the source address and the TTL
match between this packet and the one which was set.
Useful if you have problems with people spoofing their source address in order
to DoS you via this module.
--name name Name of the recent list to be used. DEFAULT used if none given.
--rsource Match/Save the source address of each packet in the recent list table (default).
--rdest Match/Save the destination address of each packet in the recent list table.
--mask netmask Netmask that will be applied to this recent list.
exemplo para bloquear a força bruta do ssh:
iptables -A INPUT -i eth0 -p tcp --syn --dport 22 -m recent --name ssh --set
iptables -A INPUT -i eth0 -p tcp --syn --dport 22 -m recent --name ssh --rcheck --seconds 30 --hitcount 2 -j DROP