O pacote VirtualBox-5.2 não é uma atualização para os pacotes existentes que você instalou, portanto, é necessário remover o último:
sudo dnf remove VirtualBox VirtualBox-server
antes de instalar VirtualBox-5.2 novamente.
Eu encontrei um problema seguindo as instruções de instalação do Fedora 26 em link
No passo 5 da página Web, o comando dnf install VirtualBox-5.2 resultou em mensagens de erro, que eu dei no final desta consulta do fórum. Como não entendi o significado dessas mensagens, continuei com o comando step 5 /usr/lib/virtualbox/vboxdrv.sh setup . Este comando falhou; o diretório /usr/lib/virtualbox/ não agora existe.
Sei por que o problema ocorreu, mas não como resolvê-lo. Há alguns meses, instalei Fedora-Workstation-Live-x86_64-26-1.5.iso e, em seguida, executei manualmente dnf install @cinnamon-desktop , seguido por dnf install VirtualBox . Naquela época, eu não executava nenhum outro comando relacionado Instalação do VirtualBox . O VirtualBox não apareceu no menu de lançamento do canela e, posteriormente, esqueci que o havia instalado. Solicito ajuda para resolver essa bagunça.
*************************** start of messages ********************************
[root@ip98-167-107-31 yum.repos.d]# dnf install VirtualBox-5.2
Last metadata expiration check: 0:17:20 ago on Fri 01 Dec 2017 01:09:48 AM PST.
dependencies resolved.
=================================================================================================================
Package Arch Version Repository Size
=================================================================================================================
Installing:
VirtualBox-5.2 x86_64 5.2.2_119230_fedora26-1 virtualbox 70 M
Transaction Summary
=================================================================================================================
Install 1 Package
Total download size: 70 M
Installed size: 154 M
Is this ok [y/N]: y
Downloading Packages:
VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64.rpm 15 MB/s | 70 MB 00:04
-----------------------------------------------------------------------------------------------------------------
Total 15 MB/s | 70 MB 00:04
warning: /var/cache/dnf/virtualbox-4917f85f98c326df/packages/VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 98ab5139: NOKEY
Importing GPG key 0x98AB5139:
Userid : "Oracle Corporation (VirtualBox archive signing key) <[email protected]>"
Fingerprint: 7B0F AB3A 13B9 0743 5925 D9C9 5442 2A4B 98AB 5139
From : https://www.virtualbox.org/download/oracle_vbox.asc
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction check error:
file /usr/bin/VBox from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/bin/VBoxTunctl from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/bin/vbox-img from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/128x128/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/16x16/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/20x20/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/32x32/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/40x40/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/48x48/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/icons/hicolor/64x64/apps/virtualbox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/mime/packages/virtualbox.xml from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-server-5.1.30-2.fc26.x86_64
file /usr/share/applications/virtualbox.desktop from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-5.1.30-2.fc26.x86_64
file /usr/share/pixmaps/VBox.png from install of VirtualBox-5.2-5.2.2_119230_fedora26-1.x86_64 conflicts with file from package VirtualBox-5.1.30-2.fc26.x86_64
Error Summary
-------------
[root@ip98-167-107-31 yum.repos.d]#
Verifique o download do pacote rpm e instale com rpm -Uvh --replacefiles