coloque todos os usuários em um grupo 'remote_users' e especifique em seu sshd.conf , que apenas os usuários desses grupos podem fazer login via ssh:
AllowGroups
This keyword can be followed by a list of group name patterns,
separated by spaces. If specified, login is allowed only for
users whose primary group or supplementary group list matches one
of the patterns. Only group names are valid; a numerical group
ID is not recognized. By default, login is allowed for all
groups. The allow/deny directives are processed in the following
order: DenyUsers, AllowUsers, DenyGroups, and finally
AllowGroups.