Acho que você está procurando a opção AllowGroups
no arquivo sshd_config
:
AllowGroups
This keyword can be followed by a list of group name patterns, separated by spaces. If specified, login is allowed only for users whose primary group
or supplementary group list matches one of the patterns. Only group names are valid; a numerical group ID is not recognized. By default, login is
allowed for all groups. The allow/deny directives are processed in the following order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups.
See PATTERNS in ssh_config(5) for more information on patterns.