NEC Cluster Batch System (laki + laki2)

The only way to start a parallel job on the compute nodes of this cluster is to use the portable batch system (Torque).

Policies for usage of the queues for the batch system

---

Submit a batch job

You will get each requested node for your exclusive usage. There are 2 methods to use the batch system:

• interactive batch jobs:

if requested resources are available, the job starts a interactive shell immediately. For interactive access the qsub command has the option -I example:

qsub -I ...

• normal batch jobs:

jobs will be started by the MOAB scheduler after passing some rules configured by the administrator (FAIRSHARE, BACKFILLING, ...).

Command for submitting a batch job request

A short explanation follows here. For detailed information, see the man pages or the documentation in the WWW.

man qsub
man pbs_resources
man pbs

Command to submit a batch job

qsub <option>

On success, the qsub command returns a request ID.

You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script. There are 2 important resources you need to specify:

1. nodes=<number of nodes>:<feature>

To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.

Available node:

feature	describes	notes
nehalem	2 quad core-CPU per node
tesla	nehalem nodes with a tesla GPU
mem12gb	nehalem node with 12GB memory
mem24g	nehalem node with 24GB memory
mem48gb	nehalem node with 48GB memory

Multi nodes can be specified using a +:

nodes=2:tesla+3:nehalem

The example above will allocate 2 nodes with feature tesla and 3 nodes with feature nehalem.

2. walltime=<time>

Usage of multi-socket nodes and multi-core cpus

The batch system takes into account the number of cpus and nodes (summarized PE) for each node when assigning resources.

---

The resource request feature nodes. If no reqeust for the number of PE per node the system assumes theat the reqeusted Wenn keine Anforderung für die Anzahl der PE auf einem Knoten angegeben wird, wird vom Batchsystem angenommen, dass die Anzahl der angefordeten Knoten gleich der Anzahl der angeforderten PE ist. Es werden dem Job dann so viele Knoten zugewiesen, bis eine ausreichende Anzahl von PE zur Verfügung steht.

Beispiel:

 qsub -l nodes=2:nehalem ./myscript

Es wird ein Knoten mit Harpertown-Prozessoren (Quadcore) zugewiesen. Der Inhalt der Datei ${PBS_NODEFILE} ist:

 node1
 node1

---

Ressourcenanforderung - Feature nodes, Option ppn. Durch Angabe der Option ppn können mehrere PE auf einem Knoten angefordert werden. Diese Option erlaubt es insbesondere Open-MPI, das in das Batchsystem integriert ist, die Prozessplatzierung so vorzunehmen, dass Prozesse mit benachbarten Ranks auch auf einem gemeinsamen bzw. benachbarten Knoten ausgeführt werden.

Beispiel:

 qsub -l nodes=2:bwgrid:ppn=2+1:bwgrid:ppn=3 ./myscript

Es werden zwei Knoten mit Harpertown-Prozesoren (Quadcore) für jeweils 2 und 1 Knoten mit 3 PE alloziiert. Damit ergibt sich folgender Inhalt für die Datei ${PBS_NODEFILE}

 node1
 node1
 node2
 node2
 node3
 node3
 node3

---

Ressourcenanforderung - Feature nodes, Option pmem. Für spezielle Anwendungsfälle kann es gewünscht sein, dass durch das Batchsystem nur ein PE je Knoten zugewiesen werden soll. Die auf dem Cluster eingesetzte Software (Torque/Maui) erfordert dafür die Zuhilfenahme eines kleinen Tricks.

Da für die Angabe nur eines PE (ppn=1) die gleiche Annahme zugrunde gelegt wird, als wenn keine Anforderung gegeben worden wäre (also wie im ersten Beispiel), werden alle PE eines Knoten verwendet. Aus diesem Grund muss durch die Verwendung einer weiteren Ressource eine Situation erzeugt werden, dass nur ein PE eines Knoten verwendet werden kann.

Am leichtesten wird dies durch eine Speicheranforderung, die mehr als die Hälfte des Speichers eines Knoten anfordert, erreicht.

Beispiel:

 qsub -l nodes=2:nehalem,pmem=11gb ./myscript

Es werden zwei Knoten mit nehalem-Prozessoren (Quadcore) zugewiesen. Diese besitzen 12 GByte RAM, so dass nur ein PE je Knoten verwendet werden kann. Der Inhalt der Datei ${PBS_NODEFILE} ist:

 node1
 node2

Standardwerte für Ressourcenanforderungen

If you don't set the resources for your job request, then you will get default resource limits for your job.

feature	value	notes
walltime	00:10:00
nodes	1
ppn	1

Please select your resource requests carefull. The higher your specified resource limits the lower the job priority. See also QueuePolicies.

To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

Verwenden einer Reservierung

Um reservierte Knoten zu verwenden, ist die Reservierung als zusätzliches Attribut zu spezifizieren:

Eine Reservierung mit dem Namen john.1 wird durch den folgenden Befehl verwendet:

qsub -l nodes=1:nehalem,walltime=1:00 -W x=FLAGS:ADVRES:john.1 testjob.cmd

Examples for PBS options in job scripts

You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two cpu and runs a
# MPI program on one node)
# The default walltime is 10min !
#
#PBS -l nodes=2:nehalem
cd $HOME/testdir
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest

VERY important is that you specify a shell in the first line of your batch script.

VERY important in case you use the openmpi module is to omit the -hostfile option. Otherwise an error will occur like

[n110402:02618] pls:tm: failed to poll for a spawned proc, return status = 17002
[n110402:02618] [0,0,0] ORTE_ERROR_LOG: In errno in file ../../../../../orte/mca/rmgr/urm/rmgr_urm.c at line 462
[n110402:02618] mpirun: spawn failed with errno=-11

If you want to use two MPI processes on each node this can be done like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on four processors (two on each node)
# The default walltime is 10min !
#
#PBS -l nodes=2:nehalem:ppn=2
cd $HOME/testdir
mpirun -np 4 -hostfile machines ./mpitest

If you need 2h wall time and one node you can use the following script:

#
# Simple PBS batch script that runs a scalar job using 2h
#
#PBS -l nodes=1:nehalem,walltime=2:00:00
cd $HOME/jobdir
./my_executable

Examples for starting batch jobs:

• Starting a script with all options specified inside the script file

  qsub <script>

• Starting a script using 3 processors and a real time of 2 hours:

  qsub -l nodes=3:nehalem,walltime=2:00:00 <script>

• Starting a script using 5 cluster nodes using 4 processors on each node with PBS Feature nehalem:

  qsub -l nodes=5:nehalem:ppn=4,walltime=2:00:00 <script>

• Starting a script using 1 cluster node with PBS Feature mem24gb and 5 processors with PBS Feature mem12gb and real job time of 1.5 hours:

  qsub -l nodes=1:mem24gb+5:mem12gb,walltime=1:30:00 <script>

• Starting a interactive batch job using 5 processors with a job real time of 300 seconds:

  qsub -I -l nodes=5:nehalem,walltime=300

  For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node.

  You can log in from the frontend or any assigned node to all other assigned nodes by

  ssh <nodename>

  If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.

• Possibilities to request 4 cluster nodes (with feature 'bwgrid' i.e. 32 processors).

  qsub -l nodes=4:bwgrid:ppn=8 <script>

  qsub -l nodes=4:bwgrid:pmem=15gb <script>

  The difference between both job submisson kinds is in the content of PBS_NODEFILE.

• Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:

  id

  Choose a valid groupname for your job:

  qsub -l nodes=5:bwgrid,walltime=300 -W group_list=<groupname>

Get the batch job status

Available commands

qstat [options]
showq [options] (showq -h for details)
nstat

For detailed informations, see man pages:

man qstat
man pbsnodes

Examples

list all batch jobs:

qstat -a

lists all batch queues with resource limit settings:

qstat -q

lists node information of a batch job ID:

qstat -n <JOB_ID>

lists detailed information of a batch job ID:

qstat -f <JOB_ID>

lists information of the PBS node status:

pbsnodes -a
pbsnodes -l

gives informatioin of PBS node and job status:

nstat

DISPLAY: X11 applications on interactive batch jobs

For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.

frontend> qsub -l nodes=2:bwgrid,walltime=300 -X -I