As fontes estão registradas em etc/fonts
.
usr@host:/etc/fonts# grep tex . -r
./conf.avail/65-fonts-texgyre.conf: <dir>/usr/share/texmf/fonts/opentype/public/tex-gyre</dir>
./conf.avail/65-fonts-texgyre.conf: <dir>/usr/share/texmf/fonts/opentype/public/tex-gyre-math</dir>
./conf.avail/65-fonts-lmodern.conf: <dir>/usr/share/texmf/fonts/opentype/public/lm</dir>
./conf.avail/65-fonts-lmodern.conf: <dir>/usr/share/texmf/fonts/opentype/public/lm-math</dir>
Basta remover os defs relativos e executar fc-cache
.