Você tem que usar apt-get upgrade
para impedir que o apt instale novos pacotes.
apt upgrade
se comporta como o comportamento que você descreveu.
Nas páginas man:
man apt-get Seção upgrade
upgrade is used to install the newest versions of all packages currently installed [...] under no circumstances are [...] packages not already installed retrieved and installed.
man apt Seção upgrade
upgrade is used to install the newest versions of all packages currently installed [...]. New packages will be installed, but existing packages will never be removed.