Two separate mpirun on a single slurm job

Question

I would like to use two sockets of a node, for two separate mpirun on single job submission, like this:

socket    ---- 0 ----                      ---- 1 ----   
core     0 1 ... 14 15                   0 1 ... 14 15
task   mpirun#1 with 16 process      mpirun#2 with 16 process     

Also, I want to run with no multithreading.

Therefore, here is what I put in the Slurm header file:

#SBATCH --nodes=1 
#SBATCH --sockets-per-node=2 
#SBATCH --cores-per-socket=16 
#SBATCH --threads-per-core=1 

Please help me to understand what should I put in []:

mpirun [something1]  python code_1.py &
mpirun [something2]  python code_2.py 

Answer 1

It turns out mpirun on its own, cannot allocate cores of a single socket exclusively, when there are more sockets available.

The solution is to use numactl along with mpirun to imply the desired policy:

mpirun -np 16   numactl --cpunodebind=0  python code_1.py &
mpirun -np 16   numactl --cpunodebind=1  python code_1.py &
wait

does the job as expected.

Answer 2

You could use -npersocket or --npersocket #number or --map-by in mpirun command.

man mpirun gives the following:

 -npersocket, --npersocket <#persocket>
              On each node, launch this many processes times the number of
              processor sockets on the node.  The -npersocket option also turns on
              the -bind-to-socket option.  (deprecated in favor of --map-by
              ppr:n:socket)

--map-by <foo>
    Map to the specified object, defaults to socket. Supported options include slot, hwthread, core, L1cache, L2cache, L3cache, socket, numa, board, node, sequential, distance, and ppr. Any object can include modifiers by adding a : and any combination of PE=n (bind n processing elements to each proc), SPAN (load balance the processes across the allocation), OVERSUBSCRIBE (allow more processes on a node than processing elements), and NOOVERSUBSCRIBE. This includes PPR, where the pattern would be terminated by another colon to separate it from the modifiers.

You could try something like this:

mpirun -n 16 -npersocket  python code_1.py &
mpirun -n 16 -npersocket  python code_2.py 

or you could try with the option --map-by ppr:16:socket. This will map 16 processes to a socket. Also to stop oversubscription use --no-oversubscribe.

mpirun -n 16 --map-by ppr:16:socket --no-oversubscribe python code_1.py &
mpirun -n 16 --map-by ppr:16:socket --no-oversubscribe python code_2.py 

You can also use -report-bindings to display the generated bindings.

If you want to use srun, then there are plenty of other options. If you are playing around with Intel MPI, then you could use I_MPI_PIN_PROCESSOR_LIST environmental variable.