[uma resposta muito tardia, mas acrescentada para outros que podem seguir]
limitar as interfaces nas quais você roda o ntpdate pode ser útil, mas parece que o seu maior problema é a falta de hardware de relógio em tempo real, portanto o enorme deslocamento inicial.
Eu sugiro que você veja o pacote fake_hwclock. A partir da descrição do pacote :
Package: fake-hwclock (0.5)
Save/restore system clock on machines without working RTC hardware
Some machines don't have a working realtime clock (RTC) unit, or no driver for the hardware that does exist. fake-hwclock is a simple set of scripts to save the kernel's current clock periodically (including at shutdown) and restore it at boot so that the system clock keeps at least close to realtime. This will stop some of the problems that may be caused by a system believing it has travelled in time back to 1970, such as needing to perform filesystem checks at every boot.
On top of this, use of NTP is still recommended to deal with the fake clock "drifting" while the hardware is halted or rebooting.