Sugiro usar o módulo pam_access.so, que permite restringir o acesso com base em ldap_group, local-group, ldap_user, local_users.
vi /etc/pam.d/common-account ** depending on which distro the client Server is
account required pam_access.so
vi /etc/security/access.conf
+ : ALL : LOCAL
- : ALL EXCEPT LDAP_GROUP1 LOCAL_GROUP1 LocalUser ldap_User : ALL
** isso negaria todos, exceto para ldap_group1, local_group1, LocalUser, ldap_User