- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -

Difference between revisions of "Phoenix Batch System"

From HLRS Platforms
Jump to navigationJump to search
 
 
(26 intermediate revisions by the same user not shown)
Line 29: Line 29:
 
</pre>
 
</pre>
  
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script. On this cluster there are 2 important resources you need to specify:
+
You have to specify the resources you need for your batch job. These resources are specified by including them in the -q option and -l option argument on the qsub command or in the PBS job script. On this cluster there are 3 important resources you need to specify:
 
<OL>
 
<OL>
 
   <li>
 
   <li>
<pre>nodes=<number of nodes>:<feature></pre>
+
<pre>-q <queuename> </pre>
 +
  <ul>
 +
If you don't specify a queuename using option -q, then your job will be submitted in the default queue <font color=red>''normal''</font>.
 +
  </ul>
 +
  </li>
 +
  <li>
 +
<pre>-l nodes=<number of nodes>:<feature></pre>
 
   <ul>
 
   <ul>
 
To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.  
 
To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.  
 
{|border="1" cellpadding="2"
 
{|border="1" cellpadding="2"
|+'''Available node features are:'''
+
|+'''Available node features and their dedicated queues are:'''
 
|-
 
|-
|'''feature'''||'''describes'''||'''notes'''
+
|'''queuename'''||'''feature'''||'''describes'''||'''notes'''
 
|-
 
|-
|''dgrid''||cpu type intel||
+
|''<font color=red>myri</font>''||''old''||old CRAY strider nodes (dual cpu, 2GHz, 4GB, myrinet)||
 
|-
 
|-
|''cell''||cpu type IBM cell||  
+
|''<font color=red>normal</font>''||''kfn4dre''|| asus boards (2 dual core cpus, 2.8GHz, 8GB, GigE)||
 +
|-
 +
|''<font color=red>normal</font>''||''rs161e4''|| transtec nodes (2 dual core cpus, 2.8GHz, 8GB, GigE)||
 +
|-
 +
|''<font color=red>normal</font>''||''rs161e5,mem8gb,...''|| transtec nodes (2 dual core cpus, 3.0GHz, 8GB/16GB/32GB/64GB, GigE)||
 +
|-
 +
|''<font color=red>ipath</font>''||''dl145g3''|| HP Proliand DL145 Server (2 dual core cpus, 2.6GHz, 8GB, Infinipath)||
 +
|-
 +
|''-''||''all''|| any node||
 
|}
 
|}
 
Multi nodes can be specified using a ''+'':
 
Multi nodes can be specified using a ''+'':
<pre>nodes=2:cell+3:dgrid</pre>
+
<pre>nodes=2:kfn4dre+3:rs161e4</pre>
The example above will allocate 2 nodes with feature ''cell'' and 3 nodes with feature ''dgrid''.  
+
The example above will allocate 2 nodes with feature ''kfn4dre'' and 3 nodes with feature ''rs161e4''.
 +
 
 +
<font color=red>'''Important:'''</font>
 +
      <ul>
 +
You can not use nodes from different queues in a single job. Its only possible to mix different nodes with different features inside one queue.<br>
 +
Example: Its not possible to use nodes of property '''old''' and property '''kfn4dre''' in one job, because the nodes are located in different queues ('''normal''' and '''myri''').
 +
      </ul>
 
   </ul>
 
   </ul>
 
   </li>
 
   </li>
Line 52: Line 72:
  
 
   <li>
 
   <li>
<pre>walltime=<time></pre>
+
<pre>-l walltime=<time></pre>
 
   </li>
 
   </li>
 
</OL>
 
</OL>
If you don't set the resources for your job request , then you will get default resource limits for your job. The specified resources are needed for job priorization. At the moment the default value for walltime is 10 minutes and the default number of nodes is 4. If your jobs exceeds the specified resource limits, Torque will stop your job! The higher your specified resource limits the lower the job priority.
+
If you don't set the resources for your job request , then you will get default resource limits for your job. The specified resources are needed for job priorization. At the moment the default value for walltime is 10 minutes and the default number of nodes is 1. If your jobs exceeds the specified resource limits, Torque will stop your job! The higher your specified resource limits the lower the job priority.
 
<br>
 
<br>
 
<br>
 
<br>
Line 69: Line 89:
 
# The default walltime is 10min !
 
# The default walltime is 10min !
 
#
 
#
#PBS -l nodes=2:dgrid
+
#PBS -q myri -l nodes=2
 
cd $HOME/testdir
 
cd $HOME/testdir
 
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest
 
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest
Line 85: Line 105:
 
# The default walltime is 10min !
 
# The default walltime is 10min !
 
#
 
#
#PBS -l nodes=2:dgrid
+
#PBS -q normal -l nodes=2
 
cd $HOME/testdir
 
cd $HOME/testdir
 
# Create your own machinefile as follows
 
# Create your own machinefile as follows
Line 97: Line 117:
 
# Simple PBS batch script that runs a scalar job using 2h
 
# Simple PBS batch script that runs a scalar job using 2h
 
#
 
#
#PBS -l nodes=1:dgrid,walltime=2:00:00
+
#PBS -q normal -l nodes=1,walltime=2:00:00
 
cd $HOME/jobdir
 
cd $HOME/jobdir
 
./my_executable
 
./my_executable
 
</pre>
 
</pre>
 +
 +
If you are using the correct and complete directives (#PBS ...) in your batch job script, then you don't need to specify options on the command qsub. The priority of options on qsub command are higher than the directives (#PBS...) options in your batch job script.
  
 
=== Examples for starting batch jobs: ===
 
=== Examples for starting batch jobs: ===
 
<UL>
 
<UL>
 
<LI>
 
<LI>
Starting a script using 3 intel nodes and a real time of 2 hours:
+
Starting a script using 3 old CRAY strider nodes and a real time of 2 hours:
<pre>qsub -l nodes=3:dgrid,walltime=2:00:00 <script></pre>
+
<pre>qsub -q myri -l nodes=3:old,walltime=2:00:00 <script></pre>
 
</li>
 
</li>
 
<li>
 
<li>
Starting a script using 5 cluster nodes with PBS Feature dgrid:
+
Starting a script using 5 cluster nodes with PBS Feature old in queue myri:
<pre>qsub -l nodes=5:dgrid,walltime=2:00:00 <script></pre>
+
<pre>qsub -q myri -l nodes=5:old,walltime=2:00:00 <script></pre>
 
</li>
 
</li>
 
<li>
 
<li>
Starting a script using 1 cluster node with PBS Feature cell and 5 cluster nodes with PBS Feature dgrid and real job time of 1.5 hours:
+
Starting a script using 1 cluster node with PBS Feature kfn4dre and 5 cluster nodes with PBS Feature rs161e4 and real job time of 1.5 hours:
<pre>qsub -l nodes=1:mem2gb+5:mem1gb,walltime=1:30:00 <script></pre>
+
<pre>qsub -l nodes=1:kfn4dre+5:rs161e4,walltime=1:30:00 <script></pre>
 
</li>
 
</li>
 
<li>
 
<li>
Starting a interactive batch job using 5 intel nodes with a job real time of 300 seconds:
+
Starting a interactive batch job using 5 nodes with a job real time of 300 seconds in default queue normal:
<pre>qsub -I -l nodes=5:dgrid,walltime=300</pre>
+
<pre>qsub -I -l nodes=5,walltime=300</pre>
 
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node
 
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node
 
</li>
 
</li>
Line 126: Line 148:
 
<pre>id</pre>
 
<pre>id</pre>
 
Choose a valid groupname for your job:
 
Choose a valid groupname for your job:
<pre>qsub -l nodes=5:dgrid,walltime=300 -W group_list=<groupname></pre>
+
<pre>qsub -q myri -l nodes=5:old,walltime=300 -W group_list=<groupname></pre>
 
</li>
 
</li>
 
</UL>
 
</UL>
Line 132: Line 154:
 
== Get the batch job status ==
 
== Get the batch job status ==
  
<li>
+
 
'''Available commands'''
+
*'''Available commands'''
  <ul>
+
<ul>
 
<pre>
 
<pre>
 
qstat [options]
 
qstat [options]
 
showq [options] (showq -h for details)
 
showq [options] (showq -h for details)
nstat
+
nstat2
 
</pre>
 
</pre>
 
For detailed informations, see man pages:
 
For detailed informations, see man pages:
Line 145: Line 167:
 
man pbsnodes
 
man pbsnodes
 
</pre>
 
</pre>
  </ul>
+
</ul>
</li>
+
 
  
<li>
+
*'''Examples'''
'''Examples'''
+
<ul>
  <ul>
 
 
list all batch jobs:
 
list all batch jobs:
 
<pre>qstat -a</pre>
 
<pre>qstat -a</pre>
Line 166: Line 187:
 
gives informatioin of PBS node and job status:
 
gives informatioin of PBS node and job status:
 
<pre>
 
<pre>
nstat
+
nstat2
 
</pre>
 
</pre>
 
+
</ul>
  
 
== DISPLAY: X11 applications on interactive batch jobs ==
 
== DISPLAY: X11 applications on interactive batch jobs ==
  
 
For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config.  
 
For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config.  
To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -V.
+
To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.
  
 
<pre>
 
<pre>
Line 181: Line 202:
 
Now you can start your interactive Batch job:
 
Now you can start your interactive Batch job:
 
<pre>
 
<pre>
frontend> qsub -l nodes=2:dgrid,walltime=300 -V -I
+
frontend> qsub -l nodes=2:asus,walltime=300 -X -I
 
</pre>
 
</pre>
  

Latest revision as of 10:00, 8 December 2008

The only way to start a parallel job on the compute nodes of this cluster is to use the portable batch system (Torque). You will get each requested node for your exclusive usage! Detailed information about the Batch system are available by the man pages 'man pbs'. There are 2 methods to use the batch system:

  • interactive batch jobs:
    if requested resources are available, the job starts a interactive shell immediately. For interactive access the qsub command has the option -I example:
    qsub -I ...


  • normal batch jobs:
    jobs will be started by the MAUI scheduler after passing some rules configured by the administrator (FAIRSHARE, BACKFILLING, ...).


Submit a batch job

Command for submitting a batch job request

 qsub <option>

On success, the qsub command returns a request ID. For detailed information, see man pages:

man pbs
man qsub
man pbs_resources

You have to specify the resources you need for your batch job. These resources are specified by including them in the -q option and -l option argument on the qsub command or in the PBS job script. On this cluster there are 3 important resources you need to specify:

  1. -q <queuename> 
      If you don't specify a queuename using option -q, then your job will be submitted in the default queue normal.
  2. -l nodes=<number of nodes>:<feature>
      To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.
      Available node features and their dedicated queues are:
      queuename feature describes notes
      myri old old CRAY strider nodes (dual cpu, 2GHz, 4GB, myrinet)
      normal kfn4dre asus boards (2 dual core cpus, 2.8GHz, 8GB, GigE)
      normal rs161e4 transtec nodes (2 dual core cpus, 2.8GHz, 8GB, GigE)
      normal rs161e5,mem8gb,... transtec nodes (2 dual core cpus, 3.0GHz, 8GB/16GB/32GB/64GB, GigE)
      ipath dl145g3 HP Proliand DL145 Server (2 dual core cpus, 2.6GHz, 8GB, Infinipath)
      - all any node

      Multi nodes can be specified using a +:

      nodes=2:kfn4dre+3:rs161e4

      The example above will allocate 2 nodes with feature kfn4dre and 3 nodes with feature rs161e4.

      Important:

        You can not use nodes from different queues in a single job. Its only possible to mix different nodes with different features inside one queue.
        Example: Its not possible to use nodes of property old and property kfn4dre in one job, because the nodes are located in different queues (normal and myri).

  3. -l walltime=<time>

If you don't set the resources for your job request , then you will get default resource limits for your job. The specified resources are needed for job priorization. At the moment the default value for walltime is 10 minutes and the default number of nodes is 1. If your jobs exceeds the specified resource limits, Torque will stop your job! The higher your specified resource limits the lower the job priority.

To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

Examples

You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on two processors (one on each node)
# The default walltime is 10min !
#
#PBS -q myri -l nodes=2
cd $HOME/testdir
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest

VERY important is that you specify a shell in the first line of your batch script.
If you want to use two MPI processes on each node this can be done like this:

#!/bin/bash
#
# Simple PBS batch script that reserves two nodes and runs a
# MPI program on four processors (two on each node)
# The default walltime is 10min !
#
#PBS -q normal -l nodes=2
cd $HOME/testdir
# Create your own machinefile as follows
cat $PBS_NODEFILE $PBS_NODEFILE|sort > machines
mpirun -np 4 -hostfile machines ./mpitest

If you need 2h wall time and one node you can use the following script:

#
# Simple PBS batch script that runs a scalar job using 2h
#
#PBS -q normal -l nodes=1,walltime=2:00:00
cd $HOME/jobdir
./my_executable

If you are using the correct and complete directives (#PBS ...) in your batch job script, then you don't need to specify options on the command qsub. The priority of options on qsub command are higher than the directives (#PBS...) options in your batch job script.

Examples for starting batch jobs:

  • Starting a script using 3 old CRAY strider nodes and a real time of 2 hours:
    qsub -q myri -l nodes=3:old,walltime=2:00:00 <script>
  • Starting a script using 5 cluster nodes with PBS Feature old in queue myri:
    qsub -q myri -l nodes=5:old,walltime=2:00:00 <script>
  • Starting a script using 1 cluster node with PBS Feature kfn4dre and 5 cluster nodes with PBS Feature rs161e4 and real job time of 1.5 hours:
    qsub -l nodes=1:kfn4dre+5:rs161e4,walltime=1:30:00 <script>
  • Starting a interactive batch job using 5 nodes with a job real time of 300 seconds in default queue normal:
    qsub -I -l nodes=5,walltime=300

    For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node

  • Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:
    id

    Choose a valid groupname for your job:

    qsub -q myri -l nodes=5:old,walltime=300 -W group_list=<groupname>

Get the batch job status

  • Available commands
    qstat [options]
    showq [options] (showq -h for details)
    nstat2
    

    For detailed informations, see man pages:

    man qstat
    man pbsnodes
    


  • Examples
    list all batch jobs:
    qstat -a

    lists all batch queues with resource limit settings:

    qstat -q

    lists node information of a batch job ID:

    qstat -n <JOB_ID>

    lists detailed information of a batch job ID:

    qstat -f <JOB_ID>

    lists information of the PBS node status:

    pbsnodes -a
    pbsnodes -l
    

    gives informatioin of PBS node and job status:

    nstat2
    

DISPLAY: X11 applications on interactive batch jobs

For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.

frontend> echo $DISPLAY
frontend:14.0

Now you can start your interactive Batch job:

frontend> qsub -l nodes=2:asus,walltime=300 -X -I

The DISPLAY variable on the allocated node where your interactive batchjob shell is running should look then similar to start your X11 application:

node01> echo $DISPLAY
frontend:14.0
node01> your_X11_application