Você tem que usar apt-get upgrade para impedir que o apt instale novos pacotes.
apt upgrade se comporta como o comportamento que você descreveu.
Nas páginas man:
man apt-get Seção upgrade
upgrade is used to install the newest versions of all packages currently installed [...] under no circumstances are [...] packages not already installed retrieved and installed.
man apt Seção upgrade
upgrade is used to install the newest versions of all packages currently installed [...]. New packages will be installed, but existing packages will never be removed.