De acordo com man sshd_config
AllowUsers:
This keyword can be followed by a list of user name patterns, separated by spaces. If specified, login is allowed only for user names that match one of the patterns. Only user names are valid; a numerical user ID is not recognized. By default, login is allowed for all users.
você deve permitir, claro, receber ssh de qualquer IP. Se você mantiver restrição de IP, restringirá a
- um usuário específico
- e do localhost ou 192.168.1.23