Tivemos um problema semelhante e ele acabou sendo relacionado ao firewall porque o JMX RMI usa duas portas, uma que você especifica e uma ligada pelo servidor aleatoriamente. De:
As you may already know if you have been confronted with this problem, the JMX RMI connector opens two ports: one is for the RMI registry, and it's the port that you usually supply with the -Dcom.sun.management.jmxremote.port= property. The other port is used to export JMX RMI connection objects. This second port is usually dynamically allocated at random. Indeed you don't need to know this port number in order to connect to the JMX agent: the only port number you need to know to connect is the RMI registry port number from which to obtain the connection objects.
Isso pode causar problemas se você tiver um firewall restritivo em algum lugar.