Esse é um problema conhecido do ESX 4.
Virtual machines display increased memory usage in vmware-guestd and vmwareservice.exe. The memory footprint of the process continues to increase until the available memory is drained and the process cannot allocate any memory. This issue is more prominent when the guest operating system has a large number of IP addresses associated with it.
A VMware lançou um patch que corrige o problema. Depois de atualizar o ESX 4, você deve atualizar o VMware Tools também na máquina virtual.