O sistema operacional fechará automaticamente todas as conexões TCP quando um processo sair 1 .
O que você está vendo provavelmente é uma conexão no estado TIME-WAIT ("... representa aguardar tempo suficiente para passar para ter certeza de que o par remoto recebeu o reconhecimento de sua solicitação de término de conexão").
Isso não deve ser um problema para os clientes, pois um host pode estabelecer várias conexões TCP com o mesmo host remoto: port.
Para um servidor, SO_REUSEADDR pode ser usado.
What exactly does SO_REUSEADDR do?
This socket option tells the kernel that even if this port is busy (in the TIME_WAIT state), go ahead and reuse it anyway. If it is busy, but with another state, you will still get an address already in use error. It is useful if your server has been shut down, and then restarted right away while sockets are still active on its port. You should be aware that if any unexpected data comes in, it may confuse your server, but while this is possible, it is not likely.
It has been pointed out that "A socket is a 5 tuple (proto, local addr, local port, remote addr, remote port). SO_REUSEADDR just says that you can reuse local addresses. The 5 tuple still must be unique!" by Michael Hunter ([email protected]). This is true, and this is why it is very unlikely that unexpected data will ever be seen by your server. The danger is that such a 5 tuple is still floating around on the net, and while it is bouncing around, a new connection from the same client, on the same system, happens to get the same remote port. This is explained by Richard Stevens in ''2.7 Please explain the TIME_WAIT state.''.
Veja também esta resposta do StackOverflow: Usando SO_REUSEADDR - O que acontece para abrir previamente o soquete?
1 Tecnicamente, quando todas as alças para essa conexão são destruídas. Vários processos podem possuir um identificador de arquivo.