Batch System PBSPro (vulcan)

Introduction

The only way to start a parallel job on the compute nodes of this system is to use the batch system. The installed batch system is based on PBSPro.

Writing a submission script is typically the most convenient way to submit your job to the batch system. You generally interact with the batch system in two ways: through options specified in job submission scripts (these are detailed below in the examples) and by using PBSPro commands on the login nodes. There are three key commands used to interact with PBSPro:

• qsub
• qstat
• qdel

Check the man page of PBSPro for more advanced commands and options

 man pbs_professional

---

Requesting Resources using batch system

You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script. The 2 important resources you have to specify are number of nodes of a specific node type and the walltime you need for this job:

1. select=<number of nodes>:<node_resource_variable=type>

To distinguish between different nodes 4 node resource variables are assigned to each node. The node_type, node_type_cpu, node_type_mem and node_type_core of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes.

Available node:

node_type	node_type_cpu	node_type_mem	node_type_core	Graphic	localscratch	linkspeed	describes	notes	# of nodes (laki)	# of nodes (laki2)
sb	SandyBridge@2.60GHz	32gb	16c			QDR	cpu type intel sandy bridge, 32GB memory	2 octa core-CPU per node	98	178
sb	SandyBridge@2.60GHz	64gb	16c			QDR	cpu type intel sandy bridge, 64GB memory	2 octa core-CPU per node	6	10
sb	SandyBridge@2.60GHz	64gb	16c			FDR	cpu type intel sandy bridge, 64GB memory	2 octa core-CPU per node	24	0
sb	SandyBridge@2.60GHz	128gb	16c			QDR	cpu type intel sandy bridge, 128GB memory	2 octa core-CPU per node	0	4 (1 shared)
hsw	Haswell@2.60GHz	128gb	20c			QDR	cpu type intel haswell, 128GB memory	2 x 10 core-CPU 2.6 ghz per node	84	0
hsw	Haswell@2.60GHz	256gb	20c			QDR	cpu type intel haswell, 256GB memory	2 x 10 core-CPU 2.6 ghz per node	4	0
hsw	Haswell@2.50GHz	128gb	24c			QDR	cpu type intel haswell, 128GB memory	2 x 12 core-CPU 2.5 ghz per node	200	0
hsw	Haswell@2.50GHz	128gb	24c			FDR	cpu type intel haswell, 128GB memory	2 x 12 core-CPU 2.5 ghz per node	144	0
hsw	Haswell@2.50GHz	256gb	24c			QDR	cpu type intel haswell, 256GB memory	2 x 12 core-CPU 2.5 ghz per node	16	0
il	Interlagos@2.6GHz	256gb	48c			QDR	interlagos node with 256GB memory		6	0
il	Interlagos@2.6GHz	256gb	48c		4TB	QDR	interlagos node with 256GB memory, 4TB local scratch disk		4	0
ib	IvyBridge@3.3Ghz	384gb	16c	Tesla K20Xm	11TB	QDR	node with intel IvyBridge@3.3Ghz, 16 cores, 384GB memory, 11TB local SSD scratch disk, Tesla K20Xm	only for single node jobs available	3	0
nh	Nehalem_EX@2.67GHz	1024gb	48c			SDR	cpu type intel nehalem, 1TByte memory,8 socket 6 core CPU's	will be shared with other jobs! Please use "qsub -q smp -l select=1:node_type=nh ..."	1 (shared)	0
nh	Nehalem@2.8GHz	144gb	8c		6TB	SDR	cpu type intel nehalem, 148GB memory, 6TB local scratch	2 octa core-CPU per node + local scratch disk	1	0
fx5800	Nehalem@2.93GHz	24gb	4c	Quadro FX 5800		SDR	Graphic node Nvidia Quadro FX 5800, 8 core intel W3540, 24GB memory	only 1 node per job! Please use "qsub -q vis -l select=1:node_type=fx5800 ..."	4	1
gtx680	Nehalem@2.53GHz	12gb	4c	GTX680		SDR	Cuda Node with GTX680, 8 core intel E5540, 12GB memory	only 1 node per job!	2	0


Multi node type job can also be specified using a +:

select=1:node_type=hsw:node_type_mem=256gb+3:node_type=hsw:node_type_mem=128gb:node_type_core=20c

The example above will allocate 1 hsw node (a 20 core ore 24 core type) with 256 GB memory and 3 hsw nodes (the 20 cores type) with 128 GB memory.


To allocate special nodes with local disk you can use the special node resource variable localscratch:

select=1:node_type=il:localscratch=4TB

Or you want hsw nodes with network interconnect FDR, then you can use the special node resource variable linkspeed:

select=4:node_type=hsw:linkspeed=FDR

2. walltime=<time>

Batch Mode

Production jobs are typically run in batch mode. Batch scripts are shell scripts containing flags and commands to be interpreted by a shell and are used to run a set of commands in sequence.

• The number of required nodes, cores, wall time and more can be determined by the parameters in the job script header with "#PBS" before any executable commands in the script.

#!/bin/bash
#PBS -N job_name
#PBS -l select=2:node_type=hsw:mpiprocs=24
#PBS -l walltime=00:20:00             
  
# Change to the direcotry that the job was submitted from
cd $PBS_O_WORKDIR

# using the INTEL MPI module
module load mpi/impi

# Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes
mpirun -np 48  ./my_mpi_executable arg1 arg2 > my_output_file 2>&1

• The job is submitted by the qsub command (all script head parameters #PBS can also be adjusted directly by qsub command options).

 qsub my_batchjob_script.pbs

• Setting qsub options on the command line will overwrite the settings given in the batch script:

 qsub -N other_name -l select=2:node_type=hsw:mpiprocs=24 -l walltime=00:20:00 my_batchjob_script.pbs

• The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.
• At the end of the execution output and error files are returned to your HOME directory
• This example will run your executable "my_mpi_executable" in parallel with 48 MPI processes (mpiprocs=24 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 24 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The mpirun example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 48 MPI processes with 24 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.

Notes

• While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the "qsub -W sandbox=PRIVATE" option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running. PBSPro creates a job-specific directory in your home directory for the staging and execution during the job is running.

Interactive batch Mode

Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:

 qsub -I -l select=2:node_type=hsw:ncpus=24:mpiprocs=24 -l walltime=00:30:00

If the requested resources are available and free (in the example above: 2 hsw nodes/24 cores each, 30 minutes, prepared for 24 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources. Now you have to use the mpirun command to launch your parallel application to the allocated compute nodes. When you are finished, enter logout to exit the batch system and return to the normal command line.

Defaults for Ressource Requests

If you don't set the resources for your job request, then you will get default resource limits for your job.

resource	value	notes
select	1
mpiprocs	1

Please select your resource requests carefull.

To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

Run job on other Account ID

There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the groupname of this project has to be passed in the group_list:

qsub -l select=1:node_type=hsw -W group_list=<groupname>

To get your available groups:

id -Gn

Warning: note that this procedure is neither applicable nor necessary for the default project (associated to the primary group), printed with "id -gn".

Usage of a Reservation

For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:

E.g. a reservation of some nodes is bound to the queue named S373:

qsub -q S373 -l select=1:node_type=hsw -l walltime=1:00 testjob.cmd

Job Arrays

Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index. This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.

Job arrays can be requested with

qsub -J <range> <my_array_jobscript>

range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.

Notes

• Job arrays cannot be interactive
• Job arrays are automatically marked as rerunnable

Examples

Examples for PBS options in job scripts

You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two exclusive SandyBridge nodes and runs only one
# MPI process on each node)
# The walltime is 10min
#
#PBS -l select=2:node_type=sb:mpiprocs=1
#PBS -l walltime=00:10:00

### go to directory where your job request was submitted 
cd $PBS_O_WORKDIR

### load for example the Intel MPI environment
module load mpi/impi
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest

Note: * you have to specify a shell in the first line of your batch script

• you have to specify the number of nodes you need and the node type
• you have to specify the walltime the job needs

VERY important in case you use the openmpi module is to omit the -hostfile option. Otherwise an error will occur like

[n110402:02618] pls:tm: failed to poll for a spawned proc, return status = 17002
[n110402:02618] [0,0,0] ORTE_ERROR_LOG: In errno in file ../../../../../orte/mca/rmgr/urm/rmgr_urm.c at line 462
[n110402:02618] mpirun: spawn failed with errno=-11

If you want to use two MPI processes on each node this can be done like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on four processors (two on each node)
# The default walltime is 10min !
#
#PBS -l nodes=2:nehalem:ppn=2
cd $HOME/testdir
mpirun -np 4 -hostfile machines ./mpitest

If you need 2h wall time and one node you can use the following script:

#
# Simple PBS batch script that runs a scalar job using 2h
#
#PBS -l nodes=1:nehalem,walltime=2:00:00
cd $HOME/jobdir
./my_executable

Examples for starting batch jobs:

• Starting a script with all options specified inside the script file

  qsub <script>

• Starting a script using 3 nodes and a real time of 2 hours:

  qsub -l nodes=3:nehalem,walltime=2:00:00 <script>

• Starting a script using 5 cluster nodes using 4 processors on each node with PBS Feature nehalem:

  qsub -l nodes=5:nehalem:ppn=4,walltime=2:00:00 <script>

• Starting a script using 1 cluster node with PBS Feature mem24gb and 5 processors with PBS Feature mem12gb and real job time of 1.5 hours:

  qsub -l nodes=1:mem24gb+5:mem12gb,walltime=1:30:00 <script>

• Starting a interactive batch job using 5 processors with a job real time of 300 seconds:

  qsub -I -l nodes=5:nehalem,walltime=300

  For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node.

  You can log in from the frontend or any assigned node to all other assigned nodes by

  ssh <nodename>

  If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.

• Possibilities to request 4 cluster nodes (with feature 'nehalem' i.e. 32 processors).

  qsub -l nodes=4:nehalem:ppn=8 <script>

  qsub -l nodes=4:nehalem:pmem=11gb <script>

  The difference between both job submisson kinds is in the content of PBS_NODEFILE.

• Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:

  id

  Choose a valid groupname for your job (abc12345 will serve as a placeholder here):

  qsub -l nodes=5:nehalem,walltime=300 -W group_list=abc12345

Get the batch job status

Please, don't run the commands qstat or showq in a iteration loop using e.g. watch. Those commands will block all other users. You can use nstat in place of qstat or showq. This command uses a SQL database which will be updated every 3 minutes.

Available commands

qstat [options]
showq [options] (showq -h for details)
nstat

For detailed informations, see man pages:

nstat -h
man qstat
man pbsnodes

Examples

list all batch jobs:

qstat -a

lists all batch queues with resource limit settings:

qstat -q

lists node information of a batch job ID:

qstat -n <JOB_ID>

lists detailed information of a batch job ID:

qstat -f <JOB_ID>

lists information of the PBS node status:

pbsnodes -a
pbsnodes -l

gives informatioin of PBS node and job status:

nstat

DISPLAY: X11 applications on interactive batch jobs

For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.

frontend> qsub -l nodes=2:nehalem,walltime=300 -X -I