- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
Phoenix Batch System
The only way to start a parallel job on the compute nodes of this cluster is to use the portable batch system (Torque). You will get each requested node for your exclusive usage! Detailed information about the Batch system are available by the man pages 'man pbs'. There are 2 methods to use the batch system:
- interactive batch jobs:
if requested resources are available, the job starts a interactive shell immediately. For interactive access the qsub command has the option -I example:
qsub -I ...
- normal batch jobs:
jobs will be started by the MAUI scheduler after passing some rules configured by the administrator (FAIRSHARE, BACKFILLING, ...).
Submit a batch job
Command for submitting a batch job request
On success, the qsub command returns a request ID. For detailed information, see man pages:
man pbs man qsub man pbs_resources
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script. On this cluster there are 2 important resources you need to specify:
nodes=<number of nodes>:<feature>
To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.
Available node features are: feature describes notes dgrid cpu type intel cell cpu type IBM cell
Multi nodes can be specified using a +:
The example above will allocate 2 nodes with feature cell and 3 nodes with feature dgrid.
If you don't set the resources for your job request , then you will get default resource limits for your job. The specified resources are needed for job priorization. At the moment the default value for walltime is 10 minutes and the default number of nodes is 4. If your jobs exceeds the specified resource limits, Torque will stop your job! The higher your specified resource limits the lower the job priority.
To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.
You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:
#!/bin/bash # # Simple PBS batch script that reserves two nodes and runs a # MPI program on two processors (one on each node) # The default walltime is 10min ! # #PBS -l nodes=2:dgrid cd $HOME/testdir mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest
VERY important is that you specify a shell in the first line of your batch script.
If you want to use two MPI processes on each node this can be done like this:
#!/bin/bash # # Simple PBS batch script that reserves two nodes and runs a # MPI program on four processors (two on each node) # The default walltime is 10min ! # #PBS -l nodes=2:dgrid cd $HOME/testdir # Create your own machinefile as follows cat $PBS_NODEFILE $PBS_NODEFILE|sort > machines mpirun -np 4 -hostfile machines ./mpitest
If you need 2h wall time and one node you can use the following script:
# # Simple PBS batch script that runs a scalar job using 2h # #PBS -l nodes=1:dgrid,walltime=2:00:00 cd $HOME/jobdir ./my_executable
Examples for starting batch jobs:
Starting a script using 3 intel nodes and a real time of 2 hours:
qsub -l nodes=3:dgrid,walltime=2:00:00 <script>
Starting a script using 5 cluster nodes with PBS Feature dgrid:
qsub -l nodes=5:dgrid,walltime=2:00:00 <script>
Starting a script using 1 cluster node with PBS Feature cell and 5 cluster nodes with PBS Feature dgrid and real job time of 1.5 hours:
qsub -l nodes=1:mem2gb+5:mem1gb,walltime=1:30:00 <script>
Starting a interactive batch job using 5 intel nodes with a job real time of 300 seconds:
qsub -I -l nodes=5:dgrid,walltime=300
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node
Starting a script which should run on other Account ID:
First you have to know which Account ID's (groupnames) are valid for your login:
Choose a valid groupname for your job:
qsub -l nodes=5:dgrid,walltime=300 -W group_list=<groupname>
Get the batch job status
qstat [options] showq [options] (showq -h for details) nstat
For detailed informations, see man pages:
man qstat man pbsnodes
list all batch jobs:
lists all batch queues with resource limit settings:
lists node information of a batch job ID:
qstat -n <JOB_ID>
lists detailed information of a batch job ID:
qstat -f <JOB_ID>
lists information of the PBS node status:
pbsnodes -a pbsnodes -l
gives informatioin of PBS node and job status:
DISPLAY: X11 applications on interactive batch jobs
For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -V.
frontend> echo $DISPLAY frontend:14.0
Now you can start your interactive Batch job:
frontend> qsub -l nodes=2:dgrid,walltime=300 -V -I
The DISPLAY variable on the allocated node where your interactive batchjob shell is running should look then similar to start your X11 application:
node01> echo $DISPLAY frontend:14.0 node01> your_X11_application