Parece que eu precisei reiniciar o cluster inteiro!
Agora os trabalhos se comportam como deveriam com cons_res .
Provavelmente tem a ver com problemas com o sistema de arquivos, conforme sugerido na documentação de slurm .
Eu sou novo no SLURM. Eu estou tentando configurar o slurm em um novo cluster.
Eu tenho 4 nós cada um tem 14 núcleos. Eu queria compartilhar os nós de maneira que cada núcleo pudesse ser executado independentemente (ou seja, o node01 pode ter 14 jobs seriais independentes acontecendo ao mesmo tempo), mas nenhum core deve executar mais de um job. Passando pelo documentaion eu percebi que precisava definir
SelectType = select/cons_res
SelectTypeParameters = CR_CORE
Então fiz isso em slurm.conf e reiniciei slurmctld . Mas agora, se eu enviar um trabalho, eu não consigo encontrar a configuração do nó, ou o trabalho acaba no estado CG.
Exemplo 1:
[sr@clstr mpitests]$ cat newHello.slrm
#!/bin/sh
#SBATCH --time=00:01:00
#SBATCH -N 1
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
module add shared openmpi/gcc/64 slurm
module load somesh/scripts/1.0
mpirun helloMPIf90
Leva a:
[sr@clstr mpitests]$ sbatch -v newHello.slrm
sbatch: defined options for program 'sbatch'
sbatch: ----------------- ---------------------
sbatch: user : 'sr'
sbatch: uid : 1003
sbatch: gid : 1003
sbatch: cwd : /home/sr/clusterTests/mpitests
sbatch: ntasks : 4 (set)
sbatch: nodes : 1-1
sbatch: jobid : 4294967294 (default)
sbatch: partition : default
sbatch: profile : 'NotSet'
sbatch: job name : 'newHello.slrm'
sbatch: reservation : '(null)'
sbatch: wckey : '(null)'
sbatch: distribution : unknown
sbatch: verbose : 1
sbatch: immediate : false
sbatch: overcommit : false
sbatch: time_limit : 1
sbatch: nice : -2
sbatch: account : (null)
sbatch: comment : (null)
sbatch: dependency : (null)
sbatch: qos : (null)
sbatch: constraints :
sbatch: geometry : (null)
sbatch: reboot : yes
sbatch: rotate : no
sbatch: network : (null)
sbatch: array : N/A
sbatch: cpu_freq_min : 4294967294
sbatch: cpu_freq_max : 4294967294
sbatch: cpu_freq_gov : 4294967294
sbatch: mail_type : NONE
sbatch: mail_user : (null)
sbatch: sockets-per-node : -2
sbatch: cores-per-socket : -2
sbatch: threads-per-core : -2
sbatch: ntasks-per-node : 4
sbatch: ntasks-per-socket : -2
sbatch: ntasks-per-core : -2
sbatch: mem_bind : default
sbatch: plane_size : 4294967294
sbatch: propagate : NONE
sbatch: switches : -1
sbatch: wait-for-switches : -1
sbatch: core-spec : NA
sbatch: burst_buffer : '(null)'
sbatch: remote command : '/home/sr/clusterTests/mpitests/newHello.slrm'
sbatch: power :
sbatch: wait : yes
sbatch: Consumable Resources (CR) Node Selection plugin loaded with argument 4
sbatch: Cray node selection plugin loaded
sbatch: Linear node selection plugin loaded with argument 4
sbatch: Serial Job Resource Selection plugin loaded with argument 4
sbatch: error: Batch job submission failed: Requested node configuration is not available
Exemplo 2:
[sr@clstr mpitests]$ cat newHello.slrm
#!/bin/sh
#SBATCH --time=00:01:00
#SBATCH -N 1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
module add shared openmpi/gcc/64 slurm
module load somesh/scripts/1.0
helloMPIf90
Leva a:
[sr@clstr mpitests]$ sbatch -v newHello.slrm
sbatch: defined options for program 'sbatch'
sbatch: ----------------- ---------------------
sbatch: user : 'sr'
sbatch: uid : 1003
sbatch: gid : 1003
sbatch: cwd : /home/sr/clusterTests/mpitests
sbatch: ntasks : 1 (set)
sbatch: nodes : 1-1
sbatch: jobid : 4294967294 (default)
sbatch: partition : default
sbatch: profile : 'NotSet'
sbatch: job name : 'newHello.slrm'
sbatch: reservation : '(null)'
sbatch: wckey : '(null)'
sbatch: distribution : unknown
sbatch: verbose : 1
sbatch: immediate : false
sbatch: overcommit : false
sbatch: time_limit : 1
sbatch: nice : -2
sbatch: account : (null)
sbatch: comment : (null)
sbatch: dependency : (null)
sbatch: qos : (null)
sbatch: constraints :
sbatch: geometry : (null)
sbatch: reboot : yes
sbatch: rotate : no
sbatch: network : (null)
sbatch: array : N/A
sbatch: cpu_freq_min : 4294967294
sbatch: cpu_freq_max : 4294967294
sbatch: cpu_freq_gov : 4294967294
sbatch: mail_type : NONE
sbatch: mail_user : (null)
sbatch: sockets-per-node : -2
sbatch: cores-per-socket : -2
sbatch: threads-per-core : -2
sbatch: ntasks-per-node : 1
sbatch: ntasks-per-socket : -2
sbatch: ntasks-per-core : -2
sbatch: mem_bind : default
sbatch: plane_size : 4294967294
sbatch: propagate : NONE
sbatch: switches : -1
sbatch: wait-for-switches : -1
sbatch: core-spec : NA
sbatch: burst_buffer : '(null)'
sbatch: remote command : '/home/sr/clusterTests/mpitests/newHello.slrm'
sbatch: power :
sbatch: wait : yes
sbatch: Consumable Resources (CR) Node Selection plugin loaded with argument 4
sbatch: Cray node selection plugin loaded
sbatch: Linear node selection plugin loaded with argument 4
sbatch: Serial Job Resource Selection plugin loaded with argument 4
Submitted batch job 108
[sr@clstr mpitests]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
108 defq newHello sr CG 0:01 1 node001
[sr@clstr mpitests]$ scontrol show job=108
JobId=108 JobName=newHello.slrm
UserId=sr(1003) GroupId=sr(1003) MCS_label=N/A
Priority=4294901756 Nice=0 Account=(null) QOS=normal
JobState=COMPLETING Reason=NonZeroExitCode Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=1:0
RunTime=00:00:01 TimeLimit=00:01:00 TimeMin=N/A
SubmitTime=2017-03-03T18:25:51 EligibleTime=2017-03-03T18:25:51
StartTime=2017-03-03T18:26:01 EndTime=2017-03-03T18:26:02 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=defq AllocNode:Sid=clstr:20260
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node001
BatchHost=node001
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=1,node=1
Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/sr/clusterTests/mpitests/newHello.slrm
WorkDir=/home/sr/clusterTests/mpitests
StdErr=/home/sr/clusterTests/mpitests/slurm-108.out
StdIn=/dev/null
StdOut=/home/sr/clusterTests/mpitests/slurm-108.out
Power=
No caso do segundo exemplo, ele permanece no estado CG até eu reinicializar o nó.
Se eu redefinir o slurm.conf para SelectType=select/linear , as coisas se comportarão normalmente como deveriam.
Não sei onde estou cometendo erro. Isso tem a ver com a configuração do slurm, ou com o meu script de submissão de tarefas de slurm, ou algo totalmente diferente.
Se alguém puder me apontar na direção certa, isso seria muito útil.
[Nota: eu originalmente postei no stackoverflow, mas o superusuário percebido pode ser um fórum melhor.]
Parece que eu precisei reiniciar o cluster inteiro!
Agora os trabalhos se comportam como deveriam com cons_res .
Provavelmente tem a ver com problemas com o sistema de arquivos, conforme sugerido na documentação de slurm .