Batch System PBSPro (Hunter)

Introduction

The only way to start a job (parallel or single node) on the compute nodes of this system is to use the batch system. The installed batch system is PBSPro.

Writing a submission script is typically the most convenient way to submit your job to the batch system. You generally interact with the batch system in two ways: through options specified in job submission scripts (details can be found below in the examples) and by using PBSPro commands on the login nodes. There are three key commands used to interact with PBSPro:

• qsub
• qstat
• qdel

Check the man page of PBSPro for more advanced commands and options with

man pbs_professional

Requesting Resources with the batch system

Resources are allocated to jobs both by explicitly requesting them and by applying specified defaults.
Jobs explicitly request resources either at the host level in chunks defined in a selection statement, or in job-wide resource requests.

Format:

• job wide request:
  qsub ... -l <resource name>=<value>

  The only resources that can be in a job-wide request are server-level or queue-level resources, such as walltime.

• selection statement:
  qsub ... -l select=<chunks>

  The only resources that can be requested in chunks are host-level resources, such as node_type and ncpus. A chunk is the smallest set of resources that will be allocated to a job. The select is one or more resource_name=value statements separated by a colon, e.g.:

  ncpus=2:node_type=mi300a

  A selection statement is of the form:

  -l select=[N:]chunk[+[N:]chunk ...]

  Note: If N is not specified, it is taken to be 1. No spaces are allowed between chunks.

  
  

Warning: all requested cluster nodes will be exclusively allocated by 1 job. The default nodes can not be shared by multiple jobs. The allocated nodes of your job will be accounted completely, even though your job uses the allocated nodes only partial

Node types

On hunter are currently 4 different node types installed. The main compute node type are the APU nodes with AMD Instinct MI300A accelerators. Additional there are also some CPU nodes available. 4 special nodes with more memory are available for special pre- and post-processing tasks. And a special node with a very large memory is also available by using a special queue. This single (smp-) node will be shared by multiple jobs/users at the same time.
So, you have to specify the resources you need for your batch jobs. These resources are specified by including them in the -l argument (selection statement and job-wide resources) on the qsub command or in the PBS job script. The 2 important resources you have to specify are number of nodes of a specific node type in the selection statement and the walltime in the job-wide resource request you need for this job:

1.  -l select=<number of nodes>:<node_resource_variable=type>:<node_resource_variable=type>... -l walltime=<time>

To distinguish between different nodes there are node resource variables assigned to each node. The node_type, node_type_storage, node_type_cpu, node_type_mem and node_type_core for each type of node. On Hunter you have to specify at least the node_type' resource variable for the CPU nodes or APU nodes. It is also possible specifying a valid combination of the resources for a specific type of node (the node_type_cpu, node_type_mem and node_type_core are only included for reasons of compatibility with the other more heterogenous clusters).

Available node types:

node_type	node_type_storage	node_type_cpu	node_type_mem	node_type_core	description	notes	# of nodes
mi300a	none (168 nodes in total), localscratch (20 nodes in total)	AMD_Instinct_MI300A_Accelerator	512gb	96c	APU nodes (compute)	available on the default queues (without specifying a queue) and queue test	188
genoa768gb64c, genoa	none (240 nodes in total), localscratch (16 nodes in total)	AMD_EPYC_9374F	768gb	64c	CPU nodes (compute)	available on the default queues (without specifying a queue) and queue test	256
genoa3tb64c	localscratch	AMD_EPYC_9354	3tb	64c	HPE (Pre-Postprocessing)	only available on special queue pre	4
genoa6tb64c	localscratch	AMD_EPYC_9354	6tb	64c	HPE (Pre-Postprocessing)	only available on special queue smp (you need additional to specify ncpus and mem). This node can be shared by multiple jobs/users!	1

A job request for APU nodes will be specified by:

qsub -l select=16:node_type=mi300a -l walltime=1:00:00

The example above will allocate 16 APU nodes with the AMD Instinct MI300A Accelerators without localscratch (NVMe) storage for 1 hour.

qsub -l select=1:node_type=mi300a:node_type_storage=localscratch -l walltime=30:00

The example above will allocate 1 APU node with the AMD Instinct MI300A Accelerators and with a localscratch (NVMe) storage for 30 minutes.

Batch Mode

Production jobs are typically run in batch mode. Batch scripts are shell scripts containing flags and commands to be interpreted by a shell and are used to run a set of commands in sequence.

• The number of required nodes, cores, walltime and more can be determined by the parameters in the job script header with "#PBS" before any executable commands in the script.

#!/bin/bash
#PBS -N job_name
#PBS -l select=2:node_type=genoa:mpiprocs=64
#PBS -l walltime=00:20:00             
  
# Change to the direcotry that the job was submitted from
cd $PBS_O_WORKDIR


# Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes
mpirun -np 128  ./my_mpi_executable arg1 arg2 > my_output_file 2>&1

• The job is submitted by the qsub command (all script head parameters #PBS can also be adjusted directly by qsub command options).

 qsub my_batchjob_script.pbs

• Setting qsub options on the command line will overwrite the settings given in the batch script:

 qsub -N other_name -l select=2:node_type=mi300a:mpiprocs=4 -l walltime=00:20:00 my_batchjob_script.pbs

• The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.
• At the end of the execution output and error files are returned to your HOME directory
• This example will run your executable "my_mpi_executable" in parallel with 8 MPI processes (mpiprocs=4 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 4 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The mpirun example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 8 MPI processes with 4 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.

Note:

While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the

qsub -koed my_batchjob_script.pbs

option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.

Interactive Batch Mode

Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:

 qsub -I -l select=2:node_type=mi300a -l walltime=00:30:00

If the requested resources are available and free (in the example above: 2 mi300a nodes, 30 minutes), then you will get a new session on the jobs head node for your requested resources. Now you can use in this new session the commands that you usually use in your job scripts to start your parallel application on the allocated nodes. When you are finished, enter logout to exit the batch system and return to the normal command line.

PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus)

In most MPI environments, the PBS_NODEFILE will be usefull to start the correct number of mpi processes on each allocated node. The jobs ${PBS_NODEFILE} contents depends on the number of MPI processes for each requested chunk. Inside a select statement of each chunk you can define a mpiprocs option (Type: integer). The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.

Example:

 qsub -l select=2:node_type=mi300a ./myscript

The batch system allocates two node of type mi300a. The file ${PBS_NODEFILE} contains:

 node1
 node2

---

If the chunk request has the option mpiprocs defined, then it is possible to allocate the defined PE's on a node. This option especially allow MPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.

select example with 2 chunk requests (seperated by '+'):

 qsub -l select=2:node_type=mi300a:mpiprocs=2+1:node_type=mi300a:mpiprocs=3 ./myscript

The batch system allocates 2 nodes of type mi300a each for 2 PE's and 1 node of type mi300a for 3 PE's. Then the file ${PBS_NODEFILE} contains:

 node1
 node1
 node2
 node2
 node3
 node3
 node3

Run job on other Account ID

There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the groupname of this project has to be passed in the group_list:

qsub -l select=1:node_type=mi300a -W group_list=<groupname>

To get your available groups:

id -Gn

Warning: note that this procedure is neither applicable nor necessary for the default project (associated to the primary group), printed with "id -gn".

Usage of a Reservation

For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:

E.g. a reservation of some nodes is bound to the queue named workday:

qsub -q workday -l select=1:node_type=mi300a -l walltime=1:00 testjob.cmd

Job Arrays

Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index. This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.

Job arrays can be requested with

qsub -J <range> -r y <my_array_jobscript>

range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.

Note:

Job arrays cannot be interactive

Job arrays needs marked as rerunable, you have to specify option -r y

Examples

Examples for PBS options in job scripts

• You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBSPro directives (#PBS ...) for the options of qsub looks like this:

  #!/bin/bash
  #
  # Simple PBS batch script that reserves two exclusive APU nodes (mi300a)
  # and runs only one MPI process on each node (in total 2 MPI processes)
  # The walltime is 10min
  #
  #PBS -l select=2:node_type=mi300a:mpiprocs=1
  #PBS -l walltime=00:10:00
  
  ### go to directory where your job request was submitted 
  cd $PBS_O_WORKDIR
  
  
  ### run you parallel application on the allocated nodes
  mpirun -np 2 ./mpitest
  

  Warning:
  

  1. you have to specify a shell in the first line of your batch script
  2. you have to specify the number of nodes you need and the node type
  3. you have to specify the walltime the job needs
  4. allocated nodes will not be shared with other jobs, even though you uses the nodes only partial


• If you want to use four MPI processes on each node this can be done like this:

  #!/bin/bash
  #
  # Simple PBS batch script that reserves two exclusive APU nodes (mi300a)
  #  and runs four MPI process on each node  (in total 8 MPI processes)
  # The walltime is 10min
  #
  #PBS -l select=2:node_type=mi300a:mpiprocs=4
  #PBS -l walltime=00:10:00
  
  ### go to directory where your job request was submitted
  cd $PBS_O_WORKDIR
  
  
  ### run you parallel application on the allocated nodes with 4 MPI processes per node with one MPI process per APU (see man mpiexec)
  mpirun -n 8 --ppn 4 --depth 24 ./mopiest
  

Warning:
If doing so, take care to distribute the processes within the nodes in a reasonable manner! Otherwise valuable resources will be unused and performance degrades!





• If you need 2h walltime and one node you can use the following script:

  #!/bin/bash
  #
  # Simple PBS batch script that runs a scalar job 
  # on 1 mi300a node using 2h
  #
  #PBS -l select=1:node_type=mi300a,walltime=2:00:00
  cd $PBS_O_WORKDIR
  ./my_executable
  

Examples for starting batch jobs:

• Starting a script with all options (#PBS <directive>) specified inside the script file

  qsub <script>



• Starting a script using 3 nodes of node_type 'mi300a' and a real time of 2 hours:

  qsub -l select=3:node_type=mi300a,walltime=2:00:00 <script>



• Starting a script using 5 cluster nodes of node_type 'mi300a' using 4 processors on each node:

  qsub -l select=5:node_type=mi300a:mpiprocs=4,walltime=2:00:00 <script>



• Starting a script using 1 cluster node of type 'genoa' and additional requesting 4 other 'mi300a' nodes using 2 MPI processes on each of this 4 nodes; real job time is 1.5 hours:

  qsub -l select=1:node_type=genoa+4:node_type=mi300a:mpiprocs=2,walltime=1:30:00 <script>



• Starting a interactive batch job using 2 mi300a nodes with a job real time of 300 seconds:

  qsub -I -l select=2:node_type=mi300a,walltime=300

  Note: For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command

  cat $PBS_NODEFILE 

  on the batch job shell or with the PBS status command

  qstat -n <jobid>

  on the master node.

  You can log in from the frontend or any assigned node to all other assigned nodes by

  export PBS_JOBID=<jobid>
  ssh <nodename>

  <jobid> is of format (number.batchserver e.g: 123456.hunter-pbs01).

  If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.


• Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:

  id

  Choose a valid groupname for your job (abc12345 will serve as a placeholder here):

  qsub -l select=2:node_type=mi300a,walltime=300 -W group_list=abc12345



• Starting a batch job on the shared node (smp) with large memory requirements: For those special node you need additional to specify the number of cores and the memory you need! (default is 1 core and 1gb memory) The nodes are configured for shared usage which means that several user jobs can run on the same node at the same time. You can only request 1 node in you batchjob and you need to specify the queue "smp".

  qsub -q smp -l select=1:ncpus=4:mem=4096gb,walltime=300

  This allocates 4 cores on the special smp node and reserve the amount of memory to 4tb on that node for your job. Job accounting is done by the requested cores and the requested memory (for that nodes operated in shared mode). Because only 1 node or 1 node type is available in this queue (smp), the node_type=genoa6tb64c does not necessarily have to be specified in the job submit.



• Starting a batch job on the pre-or postprocessing nodes with large memory requirements (< 3tb): You can only request 1 node in you batchjob and you need to specify the queue "pre".

  qsub -q pre -l select=1,walltime=300

  This allocates 1 pre-or postprocessing node for exclusive usage. Because only 1 node or 1 node type is available in this queue (pre), the node_type=genoa3tb64c does not necessarily have to be specified in the job submit.

---

---

Get the batch job status

Available commands

qstat [options]
batchstat [options]

For detailed informations, see man pages:

batchstat -h
man qstat
man pbsnodes

Examples

list all your own batch jobs:

qstat -a

list all batch jobs (anonymous)

batchstat

lists all batch queues with resource limit settings:

qstat -q

lists node information of one of your batch job ID:

qstat -n <JOB_ID>

lists detailed information of one of your batch job ID:

qstat -f <JOB_ID>

Displays estimated start time for your queued jobs

qstat -T <JOB_ID>

Displays status information for your jobs, job arrays, and subjobs:

qstat -t <JOB_ID>

lists information of the PBS node status:

pbsnodes -a
pbsnodes -l

gives informatioin of PBS node and job status:

batchstat

Note: The qstat query the status only of your own jobs! The batchstat command, on the other hand, can also provide the status of jobs if you share the same user group as the job owner.




ssh from login nodes to your allocated nodes of your job

You can only connect to nodes that belongs to you, that are, the allocated nodes of your jobs. To log in to this nodes via ssh from the login nodes (frontend nodes) you have to set the environment PBS_JOBID. First find your running jobid's and the nodes which belongs to the job:

 qstat -rnw

Next step is setting the environment PBS_JOBID on the login node:

 export PBS_JOBID=<JOB ID>

(the <JOB ID> is in form of 123456.hunter-pbs01, i.e. includes the batch server ".hunter-pbs01")
Now you are able to login form login node via ssh to the allocated nodes of your job with the corresponding jobid.




Batch Queue Policies and Limitations

Different job queues are available for efficient resource usage. The respective limitations are intended to ensure that the machine is not monopolized by individual users or groups for a longer period of time. The resources should be available to all user groups as fairly as possible.
The priorities of the jobs depends on the used queue, the number of requested nodes and the accumulated waiting (eligible) time.

In most cases users do not need to specify a job queue with the qsub command. Jobs are sorted to the right class automatically.
In general jobs with a duration up to 24 hours can be submitted by non industrial users. Some special resources like nodes with very large memory are only available for special queues which have to be specified with the qsub command. For larger jobs or special job requirements, other restrictions apply or you must consult with the project team.

Each limit settings and policies could be changed in future to adjust the cluster usage for new user requirements.

At the moment following queues and policies are defined:

route (default)

If users don't specify a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs are:

• express
• single
• normal

The destination queues have different priorities and accepts jobs only from this routing queue. Jobs cannot be submitted directly to this destination queues. Only the CPU (genoa) nodes and APU (mi300a) nodes are assigned to the destination queues.

single

This queue is intended for all single node jobs of all non industrial users.

resource	limit	note
walltime	24 hours per jobs	480 hours for all jobs in total for a single user
available nodes	1 per job (120 in total)	only single node jobs
priority		low
job queue limit	40 per user, 60 per group
job run limit	20 per user, 30 per group




normal

This queue is intended for regular multi node jobs of all non industrial users.

resource	limit	note
walltime	24 hours per job
available nodes	60 per job (256 in total by all users)	only multi node jobs
priority		low
jobs queue limit	20 jobs per user
node queue limit	96 nodes requested by all jobs per user
node run limit	70 nodes can be used in total at the same time by all jobs per user




express

This queue is intended for all industrial users.

resource	limit	note
walltime	240 hours per job	4800 hours for all jobs in total for a single user
available nodes	96 per job (256 in total)
priority		high
jobs queue limit	256 jobs per user, 512 in total
jobs run limit	64 jobs per user can run at the same time, 256 jobs can run at the same time in total
node queue limit	256 nodes requested by all jobs per user




test

This queue is for tests and development with restricted resources needs. The jobs in this queue are expected to deliver results after very short time. It's forbidden to use this queue for production jobs. Users must specify this queue. Only the CPU (genoa) nodes and APU (mi300a) nodes are assigned to this queue.

resource	limit	note
walltime	25 minutes	2 hours for all jobs in total
available nodes	16 in total
priority		very high
joblimit	1 per user (25 for ALL)




interactive

This queue is only for batch jobs in interactive batch mode which can also be for tests and development. Users can not specify this queue with the qsub submission. But interactive batch jobs will be routed to this queue automatically. Only the CPU (genoa) nodes and APU (mi300a) nodes are assigned to this queue.

resource	limit	note
walltime	8 hours
available nodes	4 nodes per user	16 in total
priority		very high
joblimit	2 per user




pre

Queue for the pre- and postprocessing nodes. There are only 4 nodes in total available. This queue "pre" is a routing queue with final destinations to the queues for industrial and non industrial jobs. The destination queues are:

• apre
• epre

The destination queues have different priorities and accepts jobs only from this routing queue. Jobs cannot be submitted directly to this destination queues. Only the nodes of node_type=genoa3tb64c are assigned to the destination queues.

apre

This queue is intended for the pre- and postprocessing jobs of all non industrial users having large memory requirements (< 3tb).

resource	limit	note
walltime	24 hours	48 hours for all jobs in total for a single user
available nodes	1 per job, 4 in total
priority		low
joblimit	20 per user (48 for ALL)




epre

This queue is intended for the pre- and postprocessing jobs of all industrial users having large memory requirements (< 3tb).

resource	limit	note
walltime	96 hours	480 hours for all jobs in total for a single user
available nodes	1 per job, 4 in total
priority		high
joblimit	48 in total







smp

Queue for the nodes that can be shared by several jobs/users. For example, a node with a memory of 6TB is available here which can be used for jobs with very high memory requirements. This queue "smp" is a routing queue with final destinations to the queues for industrial and non industrial jobs. The destination queues are:

• asmp
• esmp

The destination queues have different priorities and accepts jobs only from this routing queue. Jobs cannot be submitted directly to this destination queues. Only the smp node with node_type=genoa6tb64c is currently assigned to the destination queues. You need to specify the number of cores (ncpus=<#>) and the memory requirements (mem=<#gb>)! The defaults for the jobs are very small: 1 core (ncpus=1) and 1gb (mem=1gb). Depending on the node type, the maximum available number of cores or the available memory must also be taken into account. The node of node_type=genoa6tb64c has 64 cores and 6TB memory installed, which can also be seen from this node_type designation.

asmp

This queue is intended for jobs of all non industrial users having very large memory requirements (< 6tb).

resource	limit	note
walltime	24 hours	48 hours for all jobs in total for a single user
available nodes	1 per job
priority		low
joblimit	20 per user (48 for ALL)




esmp

This queue is intended for jobs of all industrial users having very large memory requirements (< 6tb).

resource	limit	note
walltime	96 hours	480 hours for all jobs in total for a single user
available nodes	1 per job
priority		high
joblimit	48 in total




Further information

RAM disk

For some applications that perform many temporary local I/O operations, it can make sense in justified exceptional cases to use a type of RAM disk. In this case, the RAM of the computing node is occupied. From the application's point of view, the data is written to a locally designated directory, which, however, also reduces the usable RAM of the running application when used appropriately. However, depending on how much memory application and OS will require, not all of this capacity might be available. The RAM disk is available via the following path:

$TMPDIR