Veja o comando pam_tally2 A partir do link:
EXAMPLES
Add the following line to /etc/pam.d/login to lock the account after 4
failed logins. Root account will be locked as well. The accounts will
be automatically unlocked after 20 minutes. The module does not have to
be called in the account phase because the login calls pam_setcred(3)
correctly.
auth required pam_securetty.so
auth required pam_tally2.so deny=4 even_deny_root unlock_time=1200
auth required pam_env.so
auth required pam_unix.so
auth required pam_nologin.so
account required pam_unix.so
password required pam_unix.so
session required pam_limits.so
session required pam_unix.so
session required pam_lastlog.so nowtmp
session optional pam_mail.so standard
O comando
pam_tally2 --user {user}
onde {user} é um usuário mostrará quantas falhas foram registradas.
O comando
pam_tally2 --user {user} --reset
irá redefinir o registro para {user}.
A alternativa seria fail2ban , mas isso precisa ser instalado. Ele está no USC, então sudo apt-get install fail2ban irá instalá-lo.