Batch System PBSPro (Hawk)

The batch system on Hawk is PBSPro 19.2.X. For general usage see the PBS User Guide (19.2.3)

Warning: At the moment the setup is basic. More features, testing and productive like setup will be done in February/March 2020.

Introduction

The only way to start a job (parallel or single node) on the compute nodes of this system is to use the batch system.

Writing a submission script is typically the most convenient way to submit your job to the batch system. You generally interact with the batch system in two ways: through options specified in job submission scripts (these are detailed below in the examples) and by using PBSPro commands on the login nodes. There are three key commands used to interact with PBSPro:

qsub
qstat
qdel

Check the man page of PBSPro for more advanced commands and options

 man pbs_professional

Requesting Resources using batch system

Resources are allocated to jobs both by explicitly requesting them and by applying specified defaults.
Jobs explicitly request resources either at the host level in chunks defined in a selection statement, or in job-wide resource requests.

job wide request:
```
   qsub ... -l <resource name>=<value> 
```
The only resources that can be in a job-wide request are server-level or queue-level resources, such as walltime.
selection statement:
```
   qsub ... -l select=<chunks> 
```
The only resources that can be requested in chunks are host-level resources, such as mem and ncpus. A chunk is the smallest set of resources that will be allocated to a job. It is one or more resource_name=value statements separated by a colon, e.g.:
```
ncpus=2:mem=32GB
```
```
A  selection statement is of the form:
```
```
  -l select=[N:]chunk[+[N:]chunk ...] 
```
Note: If N is not specified, it is taken to be 1. No spaces are allowed between chunks.

Warning: all requested cluster nodes will be exclusively allocated by 1 job. The default nodes can not be shared by multiple jobs. The allocated nodes of your job will be accounted completely, even though your job uses the allocated nodes only partial

Node types

You have to specify the resources you need for your batch job. These resources are specified by including them in the -l argument (selection statement and job-wide resources) on the qsub command or in the PBS job script. The 2 important resources you have to specify are number of nodes of a specific node type in the selection statement and the walltime in the job-wide resource request you need for this job:

select=<number of nodes>:<node_resource_variable=type>

node_type

node_type_cpu

node_type_mem

node_type_core

**Available node types:**
node_type	node_type_cpu	node_type_mem	node_type_core	description	notes	# of nodes
rome	AMD7742	256gb	128c	HPE Apollo9000		200
rome	AMD7742	128gb	128c	HPE Apollo9000		5432
rome	AMD7742	2048gb	128c	HPE	only available on queue smp	4
rome	AMD7742	4096gb	128c	HPE	only available on queue smp	1

A compute node type job will be specified by:

select=64:node_type=rome:node_type_mem=256gb

The example above will allocate 64 compute node with 256 GB memory.

walltime=<time>

Batch Mode

Production jobs are typically run in batch mode. Batch scripts are shell scripts containing flags and commands to be interpreted by a shell and are used to run a set of commands in sequence.

The number of required nodes, cores, wall time and more can be determined by the parameters in the job script header with "#PBS" before any executable commands in the script.

#!/bin/bash
#PBS -N job_name
#PBS -l select=2:node_type=rome:mpiprocs=128
#PBS -l walltime=00:20:00             
  
# Change to the direcotry that the job was submitted from
cd $PBS_O_WORKDIR


# Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes
mpirun -np 128  ./my_mpi_executable arg1 arg2 > my_output_file 2>&1

The job is submitted by the qsub command (all script head parameters #PBS can also be adjusted directly by qsub command options).

 qsub my_batchjob_script.pbs

Setting qsub options on the command line will overwrite the settings given in the batch script:

 qsub -N other_name -l select=2:node_type=rome:mpiprocs=128 -l walltime=00:20:00 my_batchjob_script.pbs

The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.
At the end of the execution output and error files are returned to your HOME directory
This example will run your executable "my_mpi_executable" in parallel with 256 MPI processes (mpiprocs=128 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 128 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The mpirun example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 256 MPI processes with 128 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.

Note:

While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the

qsub -koed my_batchjob_script.pbs

option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.

Interactive Batch Mode

Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:

 qsub -I -l select=2:node_type=rome:ncpus=128:mpiprocs=128 -l walltime=00:30:00

If the requested resources are available and free (in the example above: 2 rome nodes/128 cores each, 30 minutes, prepared for 128 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources. Now you have to use the mpirun command to launch your parallel application to the allocated compute nodes. When you are finished, enter logout to exit the batch system and return to the normal command line.

PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus)

In most MPI environments, the PBS_NODEFILE will be usefull to start the correct number of mpi processes on each allocated node. The jobs ${PBS_NODEFILE} contents depends on the number of MPI processes for each requested chunk. Inside a select statement of each chunk you can define a mpiprocs option (Type: integer). The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.

Example:

 qsub -l select=2:node_type=rome ./myscript

The batch system allocates two node of type rome. The file ${PBS_NODEFILE} contains:

 node1
 node2

If the chunk request has the option mpiprocs defined, then it is possible to allocate the defined PE's on a node. This option especially allow MPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.

select example with 2 chunk requests (seperated by '+'):

 qsub -l select=2:rome:mpiprocs=2+1:node_type=rome:mpiprocs=3 ./myscript

The batch system allocates 2 nodes of type rome each for 2 PE's and 1 node of type rome for 3 PE's. Then the file ${PBS_NODEFILE} contains:

 node1
 node1
 node2
 node2
 node3
 node3
 node3

Defaults for Ressource Requests

If you don't set the resources for your job request, then you will get default resource limits for your job.

resource	value	notes
select	1
mpiprocs	1
node_type	rome

Please select your resource requests carefull.

To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

Run job on other Account ID

There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the groupname of this project has to be passed in the group_list:

qsub -l select=1:node_type=rome -W group_list=<groupname>

To get your available groups:

id -Gn

Warning: note that this procedure is neither applicable nor necessary for the default project (associated to the primary group), printed with "id -gn".

Usage of a Reservation

For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:

E.g. a reservation of some nodes is bound to the queue named workday:

qsub -q workday -l select=1:node_type=rome -l walltime=1:00 testjob.cmd

Job Arrays

Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index. This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.

Job arrays can be requested with

qsub -J <range> <my_array_jobscript>

range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.

Note:

Job arrays cannot be interactive

Job arrays are automatically marked as rerunnable

Examples

Examples for PBS options in job scripts

You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBSPro directives (#PBS ...) for the options of qsub looks like this:
```
#!/bin/bash
#
# Simple PBS batch script that reserves two exclusive rome nodes
# and runs only one MPI process on each node (in total 2 MPI processes)
# The walltime is 10min
#
#PBS -l select=2:node_type=rome:mpiprocs=1
#PBS -l walltime=00:10:00

### go to directory where your job request was submitted 
cd $PBS_O_WORKDIR


### run you parallel application on the allocated nodes
mpirun -np 2 ./mpitest
```
Warning:
1. you have to specify a shell in the first line of your batch script
2. you have to specify the number of nodes you need and the node type
3. you have to specify the walltime the job needs
4. allocated nodes will not be shared with other jobs, even though you uses the nodes only partial

If you want to use four MPI processes on each node this can be done like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two exclusive rome nodes
#  and runs four MPI process on each node  (in total 8 MPI processes)
# The walltime is 10min
#
#PBS -l select=2:node_type=rome:mpiprocs=4
#PBS -l walltime=00:10:00

### go to directory where your job request was submitted
cd $PBS_O_WORKDIR


### run you parallel application on the allocated nodes
mpirun -np 8 ./mpitest

If you need 2h wall time and one node you can use the following script:

#!/bin/bash
#
# Simple PBS batch script that runs a scalar job 
# on 1 rome node using 2h
#
#PBS -l select=1:node_type=rome,walltime=2:00:00
cd $PBS_O_WORKDIR
./my_executable

Examples for starting batch jobs:

Starting a script with all options specified inside the script file
```
qsub <script>
```

Starting a script using 3 nodes of node_type 'rome' and a real time of 2 hours:
```
qsub -l select=3:node_type=rome,walltime=2:00:00 <script>
```

Starting a script using 5 cluster nodes of node_type 'rome' using 4 processors on each node:
```
qsub -l select=5:node_type=rome:mpiprocs=4,walltime=2:00:00 <script>
```

Starting a script using 1 cluster node of type 'rome' with 256GB memory and additional requesting 5 other 'rome' nodes with 128GB memory and using 128 MPI processes on each of this 5 nodes; real job time is 1.5 hours:
```
qsub -l select=1:node_type=rome:node_type_mem=256gb+5:node_type=rome:node_type_mem=128gb:mpiprocs=128,walltime=1:30:00 <script>
```

Starting a interactive batch job using 5 rome nodes with a job real time of 300 seconds:
```
qsub -I -l select=5:node_type=rome,walltime=300
```
Note: For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command
```
cat $PBS_NODEFILE 
```
on the batch job shell or with the PBS status command
```
qstat -n <jobid>
```
on the master node.
You can log in from the frontend or any assigned node to all other assigned nodes by
```
ssh <nodename>
```
If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.

Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:
```
id
```
Choose a valid groupname for your job (abc12345 will serve as a placeholder here):
```
qsub -l select=5:node_type=rome,walltime=300 -W group_list=abc12345
```

Practical Notes / Examples

Core order

On Rome-based nodes, the core id corresponds to hyperthreads and sockets as follows:

core 0 - core 63: hyperthread 0 @ socket 0
core 64 - core 127: hyperthread 0 @ socket 1
core 128 - core 191: hyperthread 1 @ socket 0
core 192 - core 256: hyperthread 1 @ socket 1

Hence, cores 128 to 256 are using the same physical resources as cores 0 to 127! Only use them if you understand the concept of hyperthreads and actually like to use them! If you do not like to use them, start a maximum of 128 threads per node only!

Pinning

We recommend to always (in hybrid as well as pure MPI jobs) use omplace to pin processes and threads to CPU cores (cf. below) in order to prevent expensive migration.

Shall I use all the available cores?

Due to limited memory bandwidth, it might be beneficial to not use all the available cores in a node. Unfortunately, you have to figure out your sweet spot by means of trial & error. While doing this, please have in mind the internal structure of the processor (cf. Processor) and try to uniformly distribute processes over architectural building blocks (i.e. CCXs, CCDs, NUMA nodes and sockets). In order to make things more easy, please use the block and stride features of omplace (cf. manpage) or use the scripts provided below to generate lists of core IDs to be passed to omplace via the -c flag if your intended placement is not possible by means of blocks & strides.

#!/usr/bin/python

#########################################################################
# Usage:   ./distribute_by_fraction.py <numerator> <denominator>        #
# Example: ./distribute_by_fraction.py 32 128                           #
#                                                                       #
# The script will then generate a list of <numerator>/<denominator>*128 #
# cores to be used, equally distributed among the available 128 cores.  #
#########################################################################

import sys

numerator   = int(sys.argv[1])
denominator = int(sys.argv[2])

core_list = ""
for offset in range(0, 127, denominator):
    for j in range(numerator):
        index = int(j*round(float(denominator)/float(numerator)))
        core_list = core_list + str(offset + index) + ","

sys.stdout.write(core_list[:-1] + "\n")

#!/usr/bin/python

#########################################################################
# Example usage: ./distribute_by_pattern.py 1 0 0 0                     #
#                                                                       #
# This will generate a list with core 0 being used, cores 1-3 not being #
# used and so on (i.e. pattern will be repeated until status of all 128 #
# cores is defined).                                                    #
#########################################################################

import sys

core_list = ""
for i in range(128):
    if sys.argv[i%(len(sys.argv) - 1) + 1] == "1":
        core_list = core_list + str(i) + ","

sys.stdout.write(core_list[:-1] + "\n")

MPI Batchjobs

pure MPI job using HPE MPI

Here is a simple pbs job script:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:node_type=rome:mpiprocs=128
#PBS -l walltime=00:20:00
 
mpirun -np 2048 ./hi.hpe

To submit the job script execute

qsub Job.hi.hpe.pbs

pure MPI job using OpenMPI

Here is a simple pbs job script:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:node_type=rome:mpiprocs=128
#PBS -l walltime=00:20:00
 
module load openmpi
mpirun -np 2048 --map-by core --bind-to core ./hi.hpe

hybrid MPI/OpenMP job using HPE MPI

To run a MPI application with 128 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:

#!/bin/bash

#PBS -N Hi_MPI_OpenMP
#PBS -l select=2:node_type=rome:mpiprocs=64:ompthreads=2
#PBS -l walltime=00:20:00
 
export OMP_NUM_THREADS=2
mpirun -np 128 omplace -nt 2 [-vv] ./hi.mpiomp

The omplace command helps with the placement of OpenMP threads within an MPI program. In the above example, the threads in a 128-process MPI program with two threads per process are placed as follows:

Rank 0, thread 0 on core 0 of socket 0 on compute node 0
Rank 0, thread 1 on core 1 of socket 0 on compute node 0
...
Rank 31, thread 1 on core 63 of socket 0 on compute node 0
Rank 32, thread 0 on core 0 of socket 1 on compute node 0
...
Rank 63, thread 1 on core 63 of socket 1 on compute node 0
Rank 64, thread 1 on core 0 of socket 0 on compute node 1
...
Rank 127, thread 1 on core 63 of socket 1 on compute node 1

The optional -vv parameter prints out the placement of the processes and threads to standard output.
Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs!

hybrid MPI/OpenMP job using HPE MPI and hyperthreads

The job described before can be run on the same physical resources with twice the number of processes respectively threads by means of hyperthreads as follows:

#!/bin/bash

#PBS -N Hi_MPI_OpenMP_HT
#PBS -l select=2:node_type=rome:mpiprocs=128:ompthreads=2
#PBS -l walltime=00:20:00
 
export OMP_NUM_THREADS=2
mpirun -np 128 omplace  -nt 2 [-vv] ./hi.mpiomp

Ranks will be placed as follows:

Rank 0, thread 0 on logical core 0 of core 0 of socket 0 on compute node 0
Rank 0, thread 1 on logical core 0 of core 1 of socket 0 on compute node 0
...
Rank 31, thread 1 on logical core 0 of core 63 of socket 0 on compute node 0
Rank 32, thread 0 on logical core 0 of core 0 of socket 1 on compute node 0
...
Rank 63, thread 1 on logical core 0 of core 63 of socket 1 on compute node 0
Rank 64, thread 0 on logical core 1 of core 0 of socket 0 on compute node 0
...
Rank 127, thread 1 on logical core 1 of core 63 of socket 1 on compute node 0
Rank 128, thread 0 on logical core 0 of core 0 of socket 0 on compute node 1
...
Rank 255, thread 1 on logical core 1 of core 63 of socket 1 on compute node 1

pure MPI job with stride > 1

If you need to let cores unused, do as follows in order to anyway uniformly distribute processes over cores:

#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=1:node_type=rome:mpiprocs=32
#PBS -l walltime=00:20:00
 
mpirun -np 32 omplace -c 0-127:st=4 ./hi.hpe

This will start processes on cores 0, 4, 8, etc., i.e. with a stride of 4 (which means having one process per CCX respectively L3 slice (cf. Processor)).

With respect to more advanced placement of processes and threads, cf.

man omplace

as well as here.

Get the batch job status

Available commands

qstat [options]
batchstat [options]

For detailed informations, see man pages:

batchstat -h
man qstat
man pbsnodes

Examples

qstat -a

list all batch jobs (anonymous)

batchstat

lists all batch queues with resource limit settings:

qstat -q

lists node information of one of your batch job ID:

qstat -n <JOB_ID>

lists detailed information of one of your batch job ID:

qstat -f <JOB_ID>

Displays estimated start time for your queued jobs

qstat -T <JOB_ID>

Displays status information for your jobs, job arrays, and subjobs:

qstat -t <JOB_ID>

lists information of the PBS node status:

pbsnodes -a
pbsnodes -l

gives informatioin of PBS node and job status:

batchstat

Further information

With respect to further details, please refer to these slides.