Cacau Batch System

The only way to start a parallel job on the compute nodes of this cluster is to use the portable batch system (Torque). You will get each requested node for your exclusive usage! Detailed information about the Batch system are available by the man pages 'man pbs'. There are 2 methods to use the batch system:

• interactive batch jobs:

if requested resources are available, the job starts a interactive shell immediately. For interactive access the qsub command has the option -I example:

qsub -I ...

• normal batch jobs:

jobs will be started by the MAUI scheduler after passing some rules configured by the administrator (FAIRSHARE, BACKFILLING, ...).

During nighttime you can use the system for long running benchmarks and/or production like runs.

Important!
The HOME filesystem on the compute nodes is located on the master node cacau.hww.de and mounted via NFS. For computation of large data, please use the local scratch area. To copy your data input/results from/to the master node, use the command scp. See also sections HOME and SCRATCH directories.

Submit a batch job

Command for submitting a batch job request

 qsub <option>

On success, the qsub command returns a request ID. For detailed information, see man pages:

man pbs
man qsub
man pbs_resources

You have to specify the resources you need for your batch job. These resources are specified by including them in the -q option and -l option argument on the qsub command or in the PBS job script. On this cluster there are 3 important resources you need to specify:

1. -q <queuename> 

   If you don't specify a queuename using option -q, then your job will be submitted in the default queue batch. In the most cases (academic users) users don't need to specify a queuename!

2. -l nodes=<number of nodes>:<feature>

   To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.

   Available node features and their dedicated queues are:

   queuename	feature	describes	notes
   batch	mem2gb, mem8gb	3.2GHz dualcore
   pp	mem128gb	2.4GHz, 8 cores, 128GByte, 1.7TB fast local scratch
   workq	mem8gb, mem16gb	3.0GHz woodcrest 4cores
   -	all	any node

   Multi nodes can be specified using a +:

   nodes=2:kfn4dre+3:rs161e4

   The example above will allocate 2 nodes with feature kfn4dre and 3 nodes with feature rs161e4.

   Important:

   You can not use nodes from different queues in a single job. Its only possible to mix different nodes with different features inside one queue.
   Example: Its not possible to use nodes of property old and property kfn4dre in one job, because the nodes are located in different queues (normal and myri).



3. -l walltime=<time>

If you don't set the resources for your job request , then you will get default resource limits for your job. The specified resources are needed for job priorization. At the moment the default value for walltime is 10 minutes and the default number of nodes is 1. If your jobs exceeds the specified resource limits, Torque will stop your job! The higher your specified resource limits the lower the job priority.

To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

Examples

You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on two processors (one on each node)
# The default walltime is 10min !
#
#PBS -q myri -l nodes=2
cd $HOME/testdir
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest

VERY important is that you specify a shell in the first line of your batch script.
If you want to use two MPI processes on each node this can be done like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on four processors (two on each node)
# The default walltime is 10min !
#
#PBS -q normal -l nodes=2
cd $HOME/testdir
# Create your own machinefile as follows
cat $PBS_NODEFILE $PBS_NODEFILE|sort > machines
mpirun -np 4 -hostfile machines ./mpitest

If you need 2h wall time and one node you can use the following script:

#
# Simple PBS batch script that runs a scalar job using 2h
#
#PBS -q normal -l nodes=1,walltime=2:00:00
cd $HOME/jobdir
./my_executable

If you are using the correct and complete directives (#PBS ...) in your batch job script, then you don't need to specify options on the command qsub. The priority of options on qsub command are higher than the directives (#PBS...) options in your batch job script.

Examples for starting batch jobs:

• Starting a script using 3 old CRAY strider nodes and a real time of 2 hours:

  qsub -q myri -l nodes=3:old,walltime=2:00:00 <script>

• Starting a script using 5 cluster nodes with PBS Feature old in queue myri:

  qsub -q myri -l nodes=5:old,walltime=2:00:00 <script>

• Starting a script using 1 cluster node with PBS Feature kfn4dre and 5 cluster nodes with PBS Feature rs161e4 and real job time of 1.5 hours:

  qsub -l nodes=1:kfn4dre+5:rs161e4,walltime=1:30:00 <script>

• Starting a interactive batch job using 5 nodes with a job real time of 300 seconds in default queue normal:

  qsub -I -l nodes=5,walltime=300

  For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node

• Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:

  id

  Choose a valid groupname for your job:

  qsub -q myri -l nodes=5:old,walltime=300 -W group_list=<groupname>

Get the batch job status

• Available commands

qstat [options]
showq [options] (showq -h for details)
nstat2

For detailed informations, see man pages:

man qstat
man pbsnodes

• Examples

list all batch jobs:

qstat -a

lists all batch queues with resource limit settings:

qstat -q

lists node information of a batch job ID:

qstat -n <JOB_ID>

lists detailed information of a batch job ID:

qstat -f <JOB_ID>

lists information of the PBS node status:

pbsnodes -a
pbsnodes -l

gives informatioin of PBS node and job status:

nstat2

DISPLAY: X11 applications on interactive batch jobs

For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.

frontend> echo $DISPLAY
frontend:14.0

Now you can start your interactive Batch job:

frontend> qsub -l nodes=2:asus,walltime=300 -X -I

The DISPLAY variable on the allocated node where your interactive batchjob shell is running should look then similar to start your X11 application:

node01> echo $DISPLAY
frontend:14.0
node01> your_X11_application