Dê uma olhada no ListenAddress no seu sshd_config.
ListenAddress
Specifies the local addresses sshd(8) should listen on. The
following forms may be used:
ListenAddress host|IPv4_addr|IPv6_addr
ListenAddress host|IPv4_addr:port
ListenAddress [host|IPv6_addr]:port
If port is not specified, sshd will listen on the address and all
prior Port options specified. The default is to listen on all
local addresses. Multiple ListenAddress options are permitted.
Additionally, any Port options must precede this option for non-
port qualified addresses.