Atualização: a solução estava mexendo com o BIOS para ativar todas as pistas PCIe para uso, além de usar Decodificação acima de 4G ativada
Estou executando várias GPUs (GTX 1060 x6) e duas não estão respondendo. Eu consultei os drivers com lspci -nnk (saída abaixo), que mostrou que 4 das GPUs tinham "driver do kernel em uso: nvidia" enquanto as outras duas não tinham nenhuma listagem de "driver do kernel". Estou executando o Ubuntu 16.04 LTS no 4.4.0-104-genérico com o CUDA 8.0 instalado para o fluxo de tensores e o driver nvidia-387 (código-fonte aberto). Alguma idéia de porque o driver do kernel não está aparecendo?
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:190f] (rev 07)
Subsystem: ASUSTeK Computer Inc. Skylake Host Bridge/DRAM Registers [1043:8694]
00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a2af]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
Kernel driver in use: xhci_hcd
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a2ba]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
Kernel driver in use: mei_me
Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a282]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
Kernel driver in use: ahci
Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a294] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:a296] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:a297] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a298] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.2 PCI bridge [0604]: Intel Corporation Device [8086:a29a] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.3 PCI bridge [0604]: Intel Corporation Device [8086:a29b] (rev f0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2c8]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a2a1]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a2f0]
Subsystem: ASUSTeK Computer Inc. Device [1043:8723]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a2a3]
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2) I219-V [1043:8672]
Kernel driver in use: e1000e
Kernel modules: e1000e
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6161]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6161]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6161]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6161]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6163]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
Atualização: a solução estava mexendo com o BIOS para ativar todas as pistas PCIe para uso, além de usar Decodificação acima de 4G ativada
Você precisa se livrar do módulo / driver 'nouveau'.
Supondo que você esteja usando o gnome, abra "Atualizações e software" - > "Drivers adicionais" e altere-o para um dos "drivers binários da NVIDIA"
Se isso não funcionar para você, a única opção é passar pelo método de lista negra documentado aqui.