Entrei em contato com o suporte do Slack e eles disseram:
Unfortunately this is a really annoying bug we have in the desktop app. When a new update is installed, it's placed in its own subdirectory within the Slack folder inside your %localappdata% folder.
From that, Windows picks this new version up as a new application and returns it back to the hidden area of your tray. This is something we're hoping to have fixed in a future release by adjusting our updates. Sadly, I don't have an ETA on when that change may occur