Depois de cavar muito mais, parece que encontrei uma solução para o problema.
A Microsoft forneceu um hotfix para o Windows Server 2012 que permite ignorar o mecanismo de filtragem básica para fluxos de multicast de entrada:
No entanto, o hotfix não está disponível para o Windows Server 2012 R2. Mas então eu encontrei este blog: link
So by now I have a solid checklist of things to configure on our servers and ensure in our applications that consume multicast messages to make sure everything runs smoothly and message loss is kept at acceptable levels. Yet, on our latest Windows Server 2012 R2 machines I had applications experiencing serious datagram loss as the amount of network traffic (in general, not just multicast) on the machine increased increased.
I researched the problem online and got the tips you’d expect: get latest NIC drivers, increase NIC receive buffer sizes, turn on offloads, turn on receive-side scaling, fine-tune receive-side scaling, increase socket buffer sizes etc. Of course, I had already tried all those things, and none of them had worked
A solução de acordo com o post do blog:
Unfortunately, the document describes an issue in Windows Server 2012 and the hotfix available there cannot be installed on Windows Server 2012 R2. Fortunately, it doesn’t have to be. You can just set the registry key and the Base Filtering Engine supports it out of the box.
New-ItemProperty HKLM:\System\CurrentControlSet\services\Tcpip\Parameters\ -Name UdpExemptPortRange -Value "XXXX-YYYY" -PropertyType MultiString -Force
Até agora, os streams correm bem novamente. Aparentemente, o BFE não pode lidar com a grande quantidade de tráfego UDP de entrada quando há também uma boa quantidade de tráfego UDP de saída.