'Slurm jobs not utilizing 100% CPU in parallelisation

I'm running an Abaqus job through Slurm with 23 tasks per cpu and 23 cpus with MPI. Using --ntasks=23 The program will after initialization launch 23 processes. The strange behaviour I have is that when checking the CPU usage with 'top' in the terminal it doesn't seem to go above around 5% usage. I want them to have 100%. If I call the abaqus executable directly with the same input parameters, the 23 cpus correctly use 100%.

If I run a job through Abaqus that doesn't launch 23 processes, the CPU usage of one process will go as high as 2000% and perform as expected.

My config looks like this

# #
MailProg=/opt/slurm-mail/bin/slurm-spool-mail.py
MpiDefault=none
# #MpiParams=ports=#-#
ProctrackType=proctrack/linuxproc
#ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
ReturnToService=1
#FastSchedule=1
# #SlurmctldPort=6817
# #SlurmdPort=6818
#
##Debug
SlurmUser=slurm

SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
SlurmdSpoolDir=/var/spool/slurmd
StateSaveLocation=/var/spool/slurmSave
# # SCHEDULING
SchedulerType=sched/builtin
SchedulerParameters=preempt_youngest_first,preempt_strict_order
PriorityType=priority/basic

SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory,CR_ONE_TASK_PER_CORE
PreemptType=preempt/partition_prio
PreemptMode=suspend,gang
SlurmctldParameters=preempt_send_user_signal

# #
JobSubmitPlugins=lua
# #
# # LOGGING AND ACCOUNTING
AccountingStorageHost=queue
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageTRES=gres/license
ClusterName=simulation
JobAcctGatherType=jobacct_gather/linux
SlurmctldLogFile=/var/log/slurmLog/slurmctld.log
SlurmSchedLogFile=/var/log/slurmLog/slurmsched.log
SlurmdLogFile=/var/log/slurmLog/slurmd.log
JobCompType=jobcomp/filetxt
# # Licenses as generic resources
GresTypes=license
# #
# # COMPUTE NODES
NodeName=nc Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 State=UNKNOWN `RealMemory=257852 Gres=license:abaqus:18` 

I was not the person who set up our Slurm system, so I'm really just looking for help on what I should look for because I'm not very familiar with it

Edit: I looked at htop instead of top and it appears that instead of using 23 cpus it's starting 23 tasks in one cpu that are all using approximately 4.3% each. Anyone know why it's doing this?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source