De acordo com a documentação do kernel:
tcp_max_orphans - INTEGER
Maximal number of TCP sockets not attached to any user file handle, held by system. If this number is exceeded orphaned connections are reset immediately and warning is printed. This limit exists only to prevent simple DoS attacks, you must not rely on this or lower the limit artificially, but rather increase it (probably, after increasing installed memory), if network conditions require more than default value, and tune network services to linger and kill such states more aggressively. Let me to remind again: each orphan eats up to ~64K of unswappable memory.
Portanto, não, isso não limita o número de conexões abertas . Em vez disso, você precisará usar o IPTables de alguma forma, porque isso é claramente uma preocupação do firewall.