Batch System PBSPro (Hawk)

The batch system on Hawk TDS is PBSPro 19.2.1. For general usage see the PBS User Guide (19.2.3)

At the moment the setup is basic and it works for the TDS only. More features, testing and productive like setup will be done in July.

Node types

There are two types of nodes installed in the TDS:

• 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
• 1 x AMD EPYC Rome (2 x 64 cores each): select with #PBS -l node_type=rome

Core order

On Naples nodes, the core id corresponds to hyperthreads and sockets as follows:

core 0 - core 31: hyperthread 0 @ socket 0
core 32 - core 63: hyperthread 0 @ socket 1
core 64 - core 95: hyperthread 1 @ socket 0
core 96 - core 127: hyperthread 1 @ socket 1

Hence, cores 64 to 128 are using the same physical resources as cores 0 to 63! Only use them if you understand the concept of hyperthreads and actually like to use them! If you do not like to use them, start a maximum of 64 threads per node.

Pinning

In case of pure MPI jobs, processes are automatically pinned to cores. In case of hybrid MPI/OpenMP jobs, use omplace to also pin threads (cf. below).

Examples

See "man pbs_resources" regarding available resources (e.g. ncpus, mpiprocs, etc.) and how to specify resources in the job script.

pure MPI job

Here is a simple pbs job script:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:mpiprocs=64
#PBS -l walltime=00:20:00
 
module load mpt
mpirun ./hi.hpe

To submit the job script execute

qsub Job.hi.hpe.pbs

hybrid MPI/OpenMP job

To run an MPI application with 64 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:

File: Job.hi.mpiomp.hpe.pbs

#!/bin/bash

#PBS -N Hi_MPI_OpenMP
#PBS -l select=2:ncpus=64:mpiprocs=32:ompthreads=2
#PBS -l walltime=00:20:00
 
module load mpt
export OMP_NUM_THREADS=2
mpirun -np 64 omplace  -nt 2 [-vv] ./hi.mpiomp

The omplace command helps with the placement of OpenMP threads within an MPI program. In the above example, the threads in a 64-process MPI program with two threads per process are placed as follows:

• Rank 0, thread 0 on core 0 of socket 0 on compute node 0
• Rank 0, thread 1 on core 1 of socket 0 on compute node 0
• ...
• Rank 15, thread 1 on core 31 of socket 0 on compute node 0
• Rank 16, thread 0 on core 0 of socket 1 on compute node 0
• ...
• Rank 31, thread 1 on core 31 of socket 1 on compute node 0
• Rank 32, thread 0 on core 0 of socket 0 on compute node 1
• ...
• Rank 63, thread 1 on core 31 of socket 1 on compute node 1

The optional -vv parameter print out the placement of the processes and threads to standard output.
Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.