- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Batch System PBSPro (Hawk): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
(48 intermediate revisions by 2 users not shown)
Line 3: Line 3:
At the moment the setup is basic and it works for the TDS only. More features, testing and productive like setup will be done in July.
At the moment the setup is basic and it works for the TDS only. More features, testing and productive like setup will be done in July.


== Examples ==


=== Single node ===
== Documentation ==
Here is an example for a job using a single node:
 
* https://www.pbsworks.com/pdfs/PBSUserGuide18.2.pdf
* https://www.pbsworks.com/pdfs/PBSRefGuide18.2.pdf
 
 
== Node types ==
 
There are two types of nodes installed in the TDS:
* 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
* 1 x AMD EPYC Rome (2 x 64 cores each):  select with #PBS -l node_type=rome
 
 
== Core order ==
 
On Rome-based nodes, the core id corresponds to hyperthreads and sockets as follows:<br>
 
core 0 - core 63: hyperthread 0 @ socket 0<br>
core 64 - core 127: hyperthread 0 @ socket 1<br>
core 128 - core 191: hyperthread 1 @ socket 0<br>
core 192 - core 256: hyperthread 1 @ socket 1<br>
 
Hence, cores 128 to 256 are using the same physical resources as cores 0 to 127! Only use them if you understand the concept of hyperthreads and ''actually'' like to use them! If you do not like to use them, start a maximum of 128 threads per node.
 
== Pinning ==
 
We recommend to ''always'' (in hybrid as well as pure MPI jobs) use omplace to pin processes and threads to CPU cores (cf. below) in order to prevent expensive migration.
 
<br>
 
== Shall I use all the available cores? ==
 
Due to limited memory bandwidth, it might be beneficial to ''not'' use all the available cores in a node. Unfortunately, you have to figure out your sweet spot by means of trial & error. While doing this, please have in mind the internal structure of the processor (cf.
[http://fs.hlrs.de/~hpcbjdic/Hawk/Hawk_Slides.pdf Tutorial Slides from HLRS Results & Review Workshop 2019]) and try to uniformly distribute processes over architectural building blocks (i.e. CCXs, CCDs, NUMA nodes and sockets).
In order to make things more easy, please use the block and stride features of omplace (cf. manpage) or use the scripts provided below to generate lists of core IDs to be passed to omplace via the -c flag if your intended placement is not possible by means of blocks & strides.
 
<pre>
#!/usr/bin/python
 
#########################################################################
# Usage:  ./distribute_by_fraction.py <numerator> <denominator>        #
# Example: ./distribute_by_fraction.py 32 128                          #
#                                                                      #
# The script will then generate a list of <numerator>/<denominator>*128 #
# cores to be used, equally distributed among the available 128 cores.  #
#########################################################################
 
import sys
 
numerator  = int(sys.argv[1])
denominator = int(sys.argv[2])
 
core_list = ""
for offset in range(0, 127, denominator):
    for j in range(numerator):
        index = int(j*round(float(denominator)/float(numerator)))
        core_list = core_list + str(offset + index) + ","
 
sys.stdout.write(core_list[:-1] + "\n")
</pre>
 
<pre>
#!/usr/bin/python
 
#########################################################################
# Example usage: ./distribute_by_pattern.py 1 0 0 0                    #
#                                                                      #
# This will generate a list with core 0 being used, cores 1-3 not being #
# used and so on (i.e. pattern will be repeated until status of all 128 #
# cores is defined).                                                    #
#########################################################################
 
import sys
 
core_list = ""
for i in range(128):
    if sys.argv[i%(len(sys.argv) - 1) + 1] == "1":
        core_list = core_list + str(i) + ","
 
sys.stdout.write(core_list[:-1] + "\n")
</pre>
 
<br>


{{command|command=
== Examples ==
qsub -l nodes=2,walltime=1:00:00
}}


The basic commands to build and run a MPI program are:
See <pre>man pbs_resources</pre> regarding available resources (e.g. ncpus, mpiprocs, etc.) and how to specify resources in the job script.
{{command|command=
module load mpt
mpicc hi.hpe.c -o hi.hpe
mpirun ./hi.hpe
}}




=== simple batch script ===
=== pure MPI job using HPE MPI ===
   
   
Here is a simple pbs job script:
Here is a simple pbs job script:
{{file|filename=Job.hi.hpe.pbs
<pre>
|content=<pre>
#!/bin/bash
#!/bin/bash


#PBS -N Hi_Thomas
#PBS -N Hi_Thomas
#PBS -l select=16:mpiprocs=64
#PBS -l select=16:node_type=naples:mpiprocs=64
#PBS -l walltime=00:20:00
#PBS -l walltime=00:20:00
   
   
module load mpt
module load mpi/hpe/mpt/2.19
mpirun ./hi.hpe
mpirun -np 1024 ./hi.hpe
</pre>
</pre>
}}


To submit the job script execute
To submit the job script execute
{{command|command=qsub Job.hi.hpe.pbs}}
{{command|command=qsub Job.hi.hpe.pbs}}


=== pure MPI job using OpenMPI ===
Here is a simple pbs job script:
<pre>
#!/bin/bash


=== simple batch script for Hybrid MPI/OpenMP application ===
#PBS -N Hi_Thomas
#PBS -l select=16:node_type=naples:mpiprocs=64
#PBS -l walltime=00:20:00
module load mpi/openmpi/4.0.1-gnu-9.1.0
mpirun -np 1024 --map-by core --bind-to core ./hi.hpe
</pre>
 
=== hybrid MPI/OpenMP job using HPE MPI ===


To run an MPI application with 64 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:
To run a MPI application with 64 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:
{{file|filename=Job.hi.mpiomp.hpe.pbs
<pre>
|content=<pre>
#!/bin/bash
#!/bin/bash


#PBS -N Hi_MPI_OpenMP
#PBS -N Hi_MPI_OpenMP
#PBS -l select=2:ncpus=64:mpiprocs=32:ompthreads=2
#PBS -l select=2:node_type=naples:mpiprocs=32:ompthreads=2
#PBS -l walltime=00:20:00
#PBS -l walltime=00:20:00
   
   
module load mpt
module load mpi/hpe/mpt/2.19
export OMP_NUM_THREADS=2
export OMP_NUM_THREADS=2
mpirun -np 64 omplace  -nt 2 [-vv] ./hi.mpiomp
mpirun -np 64 omplace  -nt 2 [-vv] ./hi.mpiomp
</pre>
</pre>
}}


The ''omplace'' command helps with the placement of OpenMP threads within an MPI program.
The ''omplace'' command helps with the placement of OpenMP threads within an MPI program.
In the above example, the threads in a 64-process MPI program with two threads per process are placed as follows:
In the above example, the threads in a 64-process MPI program with two threads per process are placed as follows:


* Rank 0, thread 0 on CPU 0 of compute node 1
* Rank 0, thread 0 on core 0 of socket 0 on compute node 0
* Rank 0, thread 1 on CPU 1 of compute node 1
* Rank 0, thread 1 on core 1 of socket 0 on compute node 0
* ...
* Rank 15, thread 1 on core 31 of socket 0 on compute node 0
* Rank 16, thread 0 on core 0 of socket 1 on compute node 0
* ...
* ...
* Rank 31, thread 1 on CPU 33 of compute node 1
* Rank 31, thread 1 on core 31 of socket 1 on compute node 0
* Rank 32, thread 0 on CPU 0 of compute node 2
* Rank 32, thread 0 on core 0 of socket 0 on compute node 1
* ...
* ...
* Rank 63, thread 0 on CPU 32 of compute node 2
* Rank 63, thread 1 on core 31 of socket 1 on compute node 1
* Rank 63, thread 1 on CPU 33 of compute node 2


The optional ''-vv'' parameter print out the placement of the processes and threads to standard output.
The optional ''-vv'' parameter print out the placement of the processes and threads to standard output.
Line 73: Line 157:
''Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.''
''Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.''


=== hybrid MPI/OpenMP job using HPE MPI and hyperthreads ===
The job described before can be run on the same physical resources with twice the number of processes respectively threads by means of hyperthreads as follows:
<pre>
#!/bin/bash


== Node types ==
#PBS -N Hi_MPI_OpenMP_HT
#PBS -l select=2:node_type=naples:mpiprocs=64:ompthreads=2
#PBS -l walltime=00:20:00
module load mpi/hpe/mpt/2.19
export OMP_NUM_THREADS=2
mpirun -np 128 omplace  -nt 2 [-vv] ./hi.mpiomp
</pre>


There are two types of nodes installed:
Ranks will be placed as follows:
* 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
* 1 x AMD EPYC Rome (2 x 64 cores each):  select with #PBS -l node_type=rome


* Rank 0, thread 0 on logical core 0 of core 0 of socket 0 on compute node 0
* Rank 0, thread 1 on logical core 0 of core 1 of socket 0 on compute node 0
* ...
* Rank 15, thread 1 on logical core 0 of core 31 of socket 0 on compute node 0
* Rank 16, thread 0 on logical core 0 of core 0 of socket 1 on compute node 0
* ...
* Rank 31, thread 1 on logical core 0 of core 31 of socket 1 on compute node 0
* Rank 32, thread 0 on logical core 1 of core 0 of socket 0 on compute node 0
* ...
* Rank 63, thread 1 on logical core 1 of core 31 of socket 1 on compute node 0
* Rank 64, thread 0 on logical core 0 of core 0 of socket 0 on compute node 1
* ...
* Rank 127, thread 1 on logical core 1 of core 31 of socket 1 on compute node 1


== Core order ==
=== pure MPI job with stride > 1 ===
If you need to let cores unused, do as follows in order to anyway uniformly distribute processes over cores:


On Naples nodes, the core id corresponds as follows to hyperthreads and sockets:<br>
<pre>
#!/bin/bash


core 0 - core 31: hyperthread 0 @ socket 0<br>
#PBS -N Hi_Thomas
core 32 - core 63: hyperthread 0 @ socket 1<br>
#PBS -l select=1:node_type=naples:mpiprocs=16
core 64 - core 95: hyperthread 1 @ socket 0<br>
#PBS -l walltime=00:20:00
core 96 - core 127: hyperthread 1 @ socket 1<br>
module load mpi/hpe/mpt/2.19
mpirun -np 16 omplace -c 0-63:st=4 ./hi.hpe
</pre>


Hence, cores 64 to 128 are using the same physical resources as cores 0 to 63! Only use them if you understand the concept of hyperthreads and ''actually'' like to use them! If you do not like to use them, start a maximum of 64 threads per node.
This will start processes on cores 0, 4, 8, etc., i.e. with a stride of 4 (which means having one process per CCX respectively L3 slice (cf. introduction slides)).

Revision as of 14:17, 11 November 2019

The batch system on Hawk TDS is PBSPro 19.2.1. For general usage see the PBS User Guide (19.2.3)

At the moment the setup is basic and it works for the TDS only. More features, testing and productive like setup will be done in July.


Documentation


Node types

There are two types of nodes installed in the TDS:

  • 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
  • 1 x AMD EPYC Rome (2 x 64 cores each): select with #PBS -l node_type=rome


Core order

On Rome-based nodes, the core id corresponds to hyperthreads and sockets as follows:

core 0 - core 63: hyperthread 0 @ socket 0
core 64 - core 127: hyperthread 0 @ socket 1
core 128 - core 191: hyperthread 1 @ socket 0
core 192 - core 256: hyperthread 1 @ socket 1

Hence, cores 128 to 256 are using the same physical resources as cores 0 to 127! Only use them if you understand the concept of hyperthreads and actually like to use them! If you do not like to use them, start a maximum of 128 threads per node.

Pinning

We recommend to always (in hybrid as well as pure MPI jobs) use omplace to pin processes and threads to CPU cores (cf. below) in order to prevent expensive migration.


Shall I use all the available cores?

Due to limited memory bandwidth, it might be beneficial to not use all the available cores in a node. Unfortunately, you have to figure out your sweet spot by means of trial & error. While doing this, please have in mind the internal structure of the processor (cf. Tutorial Slides from HLRS Results & Review Workshop 2019) and try to uniformly distribute processes over architectural building blocks (i.e. CCXs, CCDs, NUMA nodes and sockets). In order to make things more easy, please use the block and stride features of omplace (cf. manpage) or use the scripts provided below to generate lists of core IDs to be passed to omplace via the -c flag if your intended placement is not possible by means of blocks & strides.

#!/usr/bin/python

#########################################################################
# Usage:   ./distribute_by_fraction.py <numerator> <denominator>        #
# Example: ./distribute_by_fraction.py 32 128                           #
#                                                                       #
# The script will then generate a list of <numerator>/<denominator>*128 #
# cores to be used, equally distributed among the available 128 cores.  #
#########################################################################

import sys

numerator   = int(sys.argv[1])
denominator = int(sys.argv[2])

core_list = ""
for offset in range(0, 127, denominator):
    for j in range(numerator):
        index = int(j*round(float(denominator)/float(numerator)))
        core_list = core_list + str(offset + index) + ","

sys.stdout.write(core_list[:-1] + "\n")
#!/usr/bin/python

#########################################################################
# Example usage: ./distribute_by_pattern.py 1 0 0 0                     #
#                                                                       #
# This will generate a list with core 0 being used, cores 1-3 not being #
# used and so on (i.e. pattern will be repeated until status of all 128 #
# cores is defined).                                                    #
#########################################################################

import sys

core_list = ""
for i in range(128):
    if sys.argv[i%(len(sys.argv) - 1) + 1] == "1":
        core_list = core_list + str(i) + ","

sys.stdout.write(core_list[:-1] + "\n")


Examples

See

man pbs_resources

regarding available resources (e.g. ncpus, mpiprocs, etc.) and how to specify resources in the job script.


pure MPI job using HPE MPI

Here is a simple pbs job script:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:node_type=naples:mpiprocs=64
#PBS -l walltime=00:20:00
 
module load mpi/hpe/mpt/2.19
mpirun -np 1024 ./hi.hpe

To submit the job script execute

qsub Job.hi.hpe.pbs


pure MPI job using OpenMPI

Here is a simple pbs job script:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:node_type=naples:mpiprocs=64
#PBS -l walltime=00:20:00
 
module load mpi/openmpi/4.0.1-gnu-9.1.0
mpirun -np 1024 --map-by core --bind-to core ./hi.hpe

hybrid MPI/OpenMP job using HPE MPI

To run a MPI application with 64 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:

#!/bin/bash

#PBS -N Hi_MPI_OpenMP
#PBS -l select=2:node_type=naples:mpiprocs=32:ompthreads=2
#PBS -l walltime=00:20:00
 
module load mpi/hpe/mpt/2.19
export OMP_NUM_THREADS=2
mpirun -np 64 omplace  -nt 2 [-vv] ./hi.mpiomp

The omplace command helps with the placement of OpenMP threads within an MPI program. In the above example, the threads in a 64-process MPI program with two threads per process are placed as follows:

  • Rank 0, thread 0 on core 0 of socket 0 on compute node 0
  • Rank 0, thread 1 on core 1 of socket 0 on compute node 0
  • ...
  • Rank 15, thread 1 on core 31 of socket 0 on compute node 0
  • Rank 16, thread 0 on core 0 of socket 1 on compute node 0
  • ...
  • Rank 31, thread 1 on core 31 of socket 1 on compute node 0
  • Rank 32, thread 0 on core 0 of socket 0 on compute node 1
  • ...
  • Rank 63, thread 1 on core 31 of socket 1 on compute node 1

The optional -vv parameter print out the placement of the processes and threads to standard output.
Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.

hybrid MPI/OpenMP job using HPE MPI and hyperthreads

The job described before can be run on the same physical resources with twice the number of processes respectively threads by means of hyperthreads as follows:

#!/bin/bash

#PBS -N Hi_MPI_OpenMP_HT
#PBS -l select=2:node_type=naples:mpiprocs=64:ompthreads=2
#PBS -l walltime=00:20:00
 
module load mpi/hpe/mpt/2.19
export OMP_NUM_THREADS=2
mpirun -np 128 omplace  -nt 2 [-vv] ./hi.mpiomp

Ranks will be placed as follows:

  • Rank 0, thread 0 on logical core 0 of core 0 of socket 0 on compute node 0
  • Rank 0, thread 1 on logical core 0 of core 1 of socket 0 on compute node 0
  • ...
  • Rank 15, thread 1 on logical core 0 of core 31 of socket 0 on compute node 0
  • Rank 16, thread 0 on logical core 0 of core 0 of socket 1 on compute node 0
  • ...
  • Rank 31, thread 1 on logical core 0 of core 31 of socket 1 on compute node 0
  • Rank 32, thread 0 on logical core 1 of core 0 of socket 0 on compute node 0
  • ...
  • Rank 63, thread 1 on logical core 1 of core 31 of socket 1 on compute node 0
  • Rank 64, thread 0 on logical core 0 of core 0 of socket 0 on compute node 1
  • ...
  • Rank 127, thread 1 on logical core 1 of core 31 of socket 1 on compute node 1

pure MPI job with stride > 1

If you need to let cores unused, do as follows in order to anyway uniformly distribute processes over cores:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=1:node_type=naples:mpiprocs=16
#PBS -l walltime=00:20:00
 
module load mpi/hpe/mpt/2.19
mpirun -np 16 omplace -c 0-63:st=4 ./hi.hpe

This will start processes on cores 0, 4, 8, etc., i.e. with a stride of 4 (which means having one process per CCX respectively L3 slice (cf. introduction slides)).