Você pode reduzir o tempo de forma segura, mas pode encontrar problemas com conexões fechadas em redes com perda de pacotes ou jitter. Eu não começaria a sintonizar em 1 segundo, começaria em 15-30 e continuaria descendo.
Além disso, você realmente precisa corrigir seu aplicativo.
RFC 1185 tem uma boa explicação na seção 3.2:
When a TCP connection is closed, a delay of 2*MSL in TIME-WAIT state ties up the socket pair for 4 minutes (see Section 3.5 of [Postel81]. Applications built upon TCP that close one connection and open a new one (e.g., an FTP data transfer connection using Stream mode) must choose a new socket pair each time. This delay serves two different purposes:
(a) Implement the full-duplex reliable close handshake of TCP. The proper time to delay the final close step is not really related to the MSL; it depends instead upon the RTO for the FIN segments and therefore upon the RTT of the path.* Although there is no formal upper-bound on RTT, common network engineering practice makes an RTT greater than 1 minute very unlikely. Thus, the 4 minute delay in TIME-WAIT state works satisfactorily to provide a reliable full-duplex TCP close. Note again that this is independent of MSL enforcement and network speed. The TIME-WAIT state could cause an indirect performance problem if an application needed to repeatedly close one connection and open another at a very high frequency, since the number of available TCP ports on a host is less than 2**16. However, high network speeds are not the major contributor to this problem; the RTT is the limiting factor in how quickly connections can be opened and closed. Therefore, this problem will no worse at high transfer speeds. (b) Allow old duplicate segements to expire. Suppose that a host keeps a cache of the last timestamp received from each remote host. This can be used to reject old duplicate segments from earlier incarnations of the
*Note: It could be argued that the side that is sending a FIN knows what degree of reliability it needs, and therefore it should be able to determine the length of the TIME-WAIT delay for the FIN's recipient. This could be accomplished with an appropriate TCP option in FIN segments.
connection, if the timestamp clock can be guaranteed to have ticked at least once since the old conennection was open. This requires that the TIME-WAIT delay plus the RTT together must be at least one tick of the sender's timestamp clock. Note that this is a variant on the mechanism proposed by Garlick, Rom, and Postel (see the appendix), which required each host to maintain connection records containing the highest sequence numbers on every connection. Using timestamps instead, it is only necessary to keep one quantity per remote host, regardless of the number of simultaneous connections to that host.