- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Batch System PBSPro (vulcan): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
(195 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Introduction ==
== Introduction ==


The only way to start a parallel job on the compute nodes of this system is to use the batch system. The installed batch system is based on PBSPro.
Applications cannot be run directly from the login node on the compute nodes of the system.
The only way to get access to the compute nodes of the system is to use the batch system.
Therefore a batch script has to be provided, which includes all the necessary commands to run the application.
The installed batch system is [https://www.pbspro.org/ PBSPro]


Writing a submission script is typically the most convenient way to submit your job to the batch system.
You generally interact with the batch system in two ways: (1) through options specified in job submission scripts (these are detailed below in the examples) and (2) by using PBSPro commands on the login nodes.
You generally interact with the batch system in two ways: through options specified in job submission scripts (these are detailed below in the examples) and by using torque or moab commands on the login nodes. There are three key commands used to interact with torque:


* qsub
There are three key commands used to interact with PBSPro:
* qstat
; qsub : used to submit your batch script to the system to be executed as a batch job
* qdel
; qstat : to display information about submitted batch jobs
; qdel : to discard a submitted or running batch job


Check the man page of torque for more advanced commands and options
Check the man pages of the individual commands as well as the man page for PBSPro on the login node for more information:
  man pbs_professional
{{Command|command=man qsub
 
man qstat
 
man qdel
 
man pbs_professional
}}
 
If you are looking for in depth information consult the User Guide in the [https://www.altair.com/pbs-works-documentation/ PBS Pro documentation].
----
----


== Requesting Resources using batch system ==
Resources are allocated to jobs both by explicitly requesting them and by applying specified  defaults. <BR>
Jobs  explicitly request resources either at the host level in chunks defined in a selection statement, or in job-wide resource requests.
<UL>
Format:
<li> job wide request:
<pre>  qsub ... -l <resource name>=<value> </pre>
The only resources that can be in a job-wide request are server-level or queue-level resources, such as walltime.
</li>
<li> selection statement:
<pre>  qsub ... -l select=<chunks> </pre>
The only resources that can be requested in chunks are host-level resources, such  as  mem  and  ncpus.
A chunk is the smallest set of resources  that will  be  allocated  to  a  job.  It is one or more resource_name=value statements separated by a colon, e.g.:
<pre>ncpus=2:mem=32GB</pre>
A  selection statement is of the form:
<pre>  -l select=[N:]chunk[+[N:]chunk ...] </pre>
{{Note|text =
If N is not specified, it is taken to be 1.
No spaces are allowed between chunks.
}}


== Requesting Resources using batch system ==
</li>
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script.
 
The 2 important resources you have to specify are number of nodes of a specific node type and the walltime you need for this job:
</UL>
 
{{Warning|text = all requested cluster nodes will be exclusively allocated by 1 job. The default nodes can not be shared by multiple jobs. The allocated nodes of your job will be accounted completely, even though your job uses the allocated nodes only  partial}}
=== Node types ===
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l argument (selection statement and job-wide resources) on the qsub command or in the PBS job script.
The 2 important resources you have to specify are <font color=red>number of nodes of a specific node type</font> in the selection statement and the <font color=red>walltime</font> in the job-wide resource request you need for this job:
<ol>
<ol>
   <li><pre>select=<number of nodes>:<node_resource_variable=type></pre></li>
   <font color=red><li><pre>select=<number of nodes>:<node_resource_variable=type></pre></li></font>
   <ul>To distinguish between different nodes 4 node resource variables are assigned to each node. The node_type, node_type_cpu, node_type_mem and node_type_core of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes.
   <ul>To distinguish between different nodes 4 node resource variables are assigned to each node. The '''''node_type''''', '''''node_type_cpu''''', '''''node_type_mem''''' and '''''node_type_core''''' of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes. For some special nodes there are also the resources '''''localscratch''''' and '''''linkspeed''''' available.
{|border="1" cellpadding="2"
{|border="1" cellpadding="2"
|+'''Available node:'''
|+'''Available node types:'''
|-
|-
|'''node_type'''||'''node_type_cpu'''||'''node_type_mem'''||'''node_type_core'''|| '''Graphic'''||'''localscratch'''||'''linkspeed'''|||'''describes'''||'''notes'''||'''# of nodes (laki)'''||'''# of nodes (laki2)
|'''node_type'''||'''node_type_cpu'''||'''node_type_mem'''||'''node_type_core'''|| '''Graphic / Accelerator'''||'''localscratch'''||'''linkspeed'''|||'''describtion'''||'''notes'''||'''# of nodes (vulcan)'''||'''# of nodes (vulcan2)
|-
|-
|''[[sb| sb]]''|| SandyBridge@2.60GHz || 32gb || 16c || || || QDR || cpu type intel sandy bridge, 32GB memory|| 2 octa core-CPU per node||98||178
|''clx-21''|| CascadeLake@2.10GHz || 384gb || 40c || || 1.8TB || HDR-100 || Intel Xeon Gold 6230 @ 2.5GHz, CascadeLake, 384GB memory, 1.8TB /localscratch || 2 x 20 core-CPU per node || 8 || 0
|-
|-
|''[[sb| sb]]''|| SandyBridge@2.60GHz || 64gb || 16c || || || QDR || cpu type intel sandy bridge, 64GB memory|| 2 octa core-CPU per node||6||10
|''clx-25''|| CascadeLake@2.50GHz || 384gb || 40c || || || HDR-100 || Intel Xeon Gold 6248 @ 2.5GHz, CascadeLake, 384GB memory || 2 x 20 core-CPU per node || 84 || 0
|-
|-
|''[[sb| sb]]''|| SandyBridge@2.60GHz || 64gb || 16c || || || FDR || cpu type intel sandy bridge, 64GB memory|| 2 octa core-CPU per node||24||0
|''[[hsw|hsw]]''|| Haswell@2.60GHz || 128gb || 20c || || || QDR || Intel Xeon E5-2660v3 @ 2.60GHz, Haswell, 128GB memory || 2 x 10 core-CPU per node || 76 || 0
|-
|-
|''[[sb| sb]]''|| SandyBridge@2.60GHz || 128gb || 16c || || || QDR || cpu type intel sandy bridge, 128GB memory|| 2 octa core-CPU per node||0||4 (1 shared)
|''[[hsw|hsw]]''|| Haswell@2.60GHz || 256gb || 20c || || || QDR || Intel Xeon E5-2660v3 @ 2.60GHz, Haswell, 256GB memory || 2 x 10 core-CPU per node || 4 || 0
|-
|-
|''[[hsw|hsw]]''|| Haswell@2.60GHz || 128gb || 20c || || || QDR || cpu type intel haswell, 128GB memory|| 2 x 10 core-CPU 2.6 ghz per node||84||0
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 128gb || 24c || || || QDR || Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 128GB memory || 2 x 12 core-CPU per node || 152 || 192 (1 is in shared mode)
|-
|-
|''[[hsw|hsw]]''|| Haswell@2.60GHz || 256gb || 20c || || || QDR || cpu type intel haswell, 256GB memory|| 2 x 10 core-CPU 2.6 ghz per node||4||0
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 128gb || 24c || || || FDR || Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 128GB memory || 2 x 12 core-CPU per node || 0 || 0
|-
|-
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 128gb || 24c || || || QDR || cpu type intel haswell, 128GB memory|| 2 x 12 core-CPU 2.5 ghz per node||200||0
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 256gb || 24c || || || QDR || Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 256GB memory || 2 x 12 core-CPU per node || 16 || 0
|-
|-
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 128gb || 24c || || || FDR || cpu type intel haswell, 128GB memory|| 2 x 12 core-CPU 2.5 ghz per node||144||0
|''skl''|| Skylake@2.0GHz || 192gb || 40c || || || EDR || Intel Xeon Gold 6138 @ 2.00GHz, Skylake, 92GB memory || 2 x 20 core-CPU per node || 88 || 0
|-
|-
|''[[hsw|hsw]]''|| Haswell@2.50GHz || 256gb || 24c || || || QDR|| cpu type intel haswell, 256GB memory|| 2 x 12 core-CPU 2.5 ghz per node||16||0
|''smp''|| Skylake@2.40GHz || 1536gb || 40c || || || FDR || Intel Gold 6148 @ 2.40GHz, Skylake, 1.5TByte memory || 2 x 20 core-CPU per node<br>will be shared with other jobs!<font color=red> Please use <tt>"qsub -q smp -l select=1:node_type=smp ..." </tt></font> || 1 (shared)|| 0
|-
|-
|''[[mem256gb|mem256gb, scratch4tb ]]''||interlagos node with 256GB memory, 4 nodes have additional a 4TB local scratch disk|| ||10||0
|''[[NEC_Aurora_HW| aurora]]''|| Skylake@2.60GHz || 192gb || 24c || 8x NEC Aurora vector CPUs|| || EDR || Intel Xeon Gold 6126 | 2.60GHz, Skylake, 192GB memory, 8 NEC Aurora TSUBASA nodes || 2 x 12 core-CPU per node<br><font color=red> Please use <tt>"qsub -q vector -l select=1:node_type=aurora ..." </tt></font> || 8 || 0
|-
|-
|''mem384gb, scratch11tb,k20xm ''||node with intel IvyBridge@3.3Ghz, 16 cores, 384GB memory, 11TB local SSD scratch disk, Tesla K20Xm|| only for single node jobs available ||3||0
|''clx-ai''|| CascadeLake@2.60GHz || 768gb || 36c || 8x Tesla V100 SXM2 32GB || 7.3TB || 4x HDR-100 || Intel Xeon Gold 6240 @ 2.6GHz, CascadeLake, 768GB memory, 7.3TB /localscratch, 220GB /tmp on SSD, 8x Tesla V100 SXM2 32GB || 2 x 18 core-CPU + 8 GPU's per node || 1 || 0
|-
|-
|''smp''||cpu type intel nehalem, 1TByte memory,8 socket 6 core CPU's|| will be shared with other jobs!<font color=red> Please use <tt>"qsub -q smp -l nodes=1:smp..." </tt></font>||1
|''k20xm''|| IvyBridge@3.3Ghz || 384gb || 16c || Nvidia Tesla K20Xm || 11TB || QDR || Intel E5-2667v2 @ 3.30GHz, IvyBridge, 384GB memory, 11TB local SSD scratch disk, Nvidia Tesla K20Xm || 2 x 8 core-CPU per node<br>only for single node jobs available || 3 || 0
|-
|-
|''mem144gb, scratch6tb, scratch2tb''||cpu type intel nehalem, 148GB memory, 2TB local scratch (1 nodes) or 6TB local scratch (1 nodes)|| 2 octa core-CPU per node + local scratch disk||1
|''p100''|| Broadwell@3.00GHz || 1024gb || 24c || Nvidia Tesla P100 || 11TB || QDR || Cuda Node with Nvidia Tesla P100<br>Intel Xeon E5-2687Wv4 @ 3.0GHz, Broadwell, 1TB memory, 11TB local SSD scratch disk || 2 x 8 core-CPU per node<br>reserved for special users! <font color=red> Please use <tt>"qsub -q R_i1 -l select=1:node_type=p100 ..." </tt></font> || 3 || 0
|-
|-
|''vis''||Graphic node Nvidia Quadro FX 5800, 8 core intel W3540, 24GB memory|| only 1 node per job! <font color=red> Please use <tt>"qsub -q vis -l nodes=1:vis..." </tt></font>||3||1
|''visamd''|| Skylake@2.60GHz || 96gb || 8c || AMD Radeon PRO WX 8200 || || HDR-100 || Graphic Node with AMD Radeon PRO WX 8200<br> Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory || 2 x 4 core-CPU per node<br>only 1 node per job!  <font color=red> Please use <tt>"qsub -q vis -l select=1:node_type=visamd ..." </tt></font> || 6 || 0
|-
|''visnv''|| Skylake@2.60GHz || 96gb || 8c || Nvidia Quadro RTX 4000 || || HDR-100 || Graphic Node with Nvidia Quadro RTX 4000<br>Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory || 2 x 6 core-CPU per node<br>only 1 node per job! (1 node is reserved for special users) <font color=red> Please use <tt>"qsub -q vis -l select=1:node_type=visnv ..." </tt></font> || 2 || 1
|-
|''visp100''|| Broadwell@3.20GHz || 256gb || 16c || Nvidia Tesla P100 || || FDR || Cuda Node with Nvidia Tesla P100<br>Intel Xeon E5-2667v4 @ 3.2GHz, Broadwell, 256GB memory || 2 x 8 core-CPU per node<br>only 1 node per job! <font color=red> Please use <tt>"qsub -q vis -l select=1:node_type=visp100 ..." </tt></font> || 10 || 0
|-
|-
|''gtx680''||Cuda Node with GTX680, 8 core intel E5540, 12GB memory|| only 1 node per job! ||2||0
|}
|}
Multi nodes can be specified using a ''+'': <pre>nodes=2:tesla+3:nehalem</pre> The example above will allocate 2 nodes with feature ''tesla'' and 3 nodes with feature ''nehalem''.</ul>
<li><pre>walltime=<time></pre></li>
</OL>




'''Multi node type job can also be specified using a ''+'':'''
<ul>
<pre>select=1:node_type=hsw:node_type_mem=256gb+3:node_type=hsw:node_type_mem=128gb:node_type_core=20c</pre>
The example above will allocate 1 hsw node (a 20 core ore 24 core type) with 256 GB memory and 3 hsw nodes (the 20 cores type) with 128 GB memory.
</ul>
To allocate special nodes with local disk you can use the special node resource variable '''''localscratch''''':
<ul>
<pre>select=1:node_type=clx-21:localscratch=1800GB</pre>
</ul>
Or you want hsw nodes with network interconnect QDR, then you can use the special node resource variable  '''''linkspeed''''':
<ul>
<pre>select=4:node_type=hsw:linkspeed=QDR</pre>
</ul>
</ul>
<font color=red><li><pre>walltime=<time></pre></li></font>
</OL>


=== Batch Mode ===
=== Batch Mode ===
Line 74: Line 130:
cd $PBS_O_WORKDIR
cd $PBS_O_WORKDIR


module load mpi/your_mpi_version_for_your_application
# using the INTEL MPI module
module load mpi/impi


# Launch the parallel job to the allocated compute nodes
# Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes
mpirun opt1 opt2 ./my_mpi_executable arg1 arg2 > my_output_file 2>&1
mpirun -np 48 ./my_mpi_executable arg1 arg2 > my_output_file 2>&1
</pre>
</pre>
* The job is submitted by the '''qsub''' command (all script head parameters #PBS can also be adjusted directly by '''qsub''' command options).  
* The job is submitted by the '''qsub''' command (all script head parameters #PBS can also be adjusted directly by '''qsub''' command options).  
Line 85: Line 142:
* The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.   
* The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.   
* At the end of the execution output and error files are returned to your HOME directory
* At the end of the execution output and error files are returned to your HOME directory
* This example will run your executable "my_mpi_executable" in parallel with 48 MPI processes. The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 24 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job.  The '''mpirun''' example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 48 MPI processes with 24 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.
* This example will run your executable "my_mpi_executable" in parallel with 48 MPI processes (mpiprocs=24 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 24 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job.  The '''mpirun''' example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 48 MPI processes with 24 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.
 
{{Note|text =
<li> While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the
<pre>qsub -koed my_batchjob_script.pbs</pre>
option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.</li>
}}


=== Interactive batch Mode ===
=== Interactive batch Mode ===
Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:
  qsub -I -l select=2:node_type=hsw:ncpus=24:mpiprocs=24 -l walltime=00:30:00


== Submit a batch job ==
If the requested resources are available and free (in the example above: 2 hsw nodes/24 cores each, 30 minutes, prepared for 24 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources.
You will get each requested node for your exclusive usage. There are 2 methods to use the batch system:
Now you have to use the '''mpirun''' command to launch your parallel application to the allocated compute nodes.
# interactive batch jobs:
When you are finished, enter '''logout''' to exit the batch system and return to the normal command line.
<ul>
if requested resources are available, the job starts a interactive shell immediately. For interactive access the qsub command has the option ''-I'' example:
<pre>qsub -I ...</pre>
</ul>
# normal batch jobs:
<ul>
jobs will be started by the MOAB scheduler after passing some rules configured by the administrator (FAIRSHARE, BACKFILLING, ...).
</ul>


=== Command for submitting a batch job request ===
A short explanation follows here. For detailed information, see the man pages or the [http://www.adaptivecomputing.com/resources/docs/torque/help.htm latest documentation in the WWW].
man qsub
man pbs_resources
man pbs


Command to submit a batch job
=== PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus) ===
qsub <option>
In most MPI environments, the PBS_NODEFILE will be usefull to start the correct number of mpi processes on each allocated node.
On success, the qsub command returns a request ID.
The jobs <tt>${PBS_NODEFILE}</tt> contents depends on the number of MPI processes for each requested chunk.
 
Inside a select statement of each chunk you can define a '''mpiprocs''' option (Type: integer).
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l option argument on the qsub command or in the PBS job script. There are 2 important resources you need to specify:
The number of lines  in PBS_NODEFILE  is the sum of the values of mpiprocs for all chunks requested by the job.  For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.
<ol>
  <li><pre>nodes=<number of nodes>:<feature></pre></li>
  <ul>To distinguish between different nodes, features are assigned to each node. These features describe the properties of each node. Please do only use exactly 1 feature for each node type.
{|border="1" cellpadding="2"
|+'''Available node:'''
|-
|'''feature'''||'''describes'''||'''notes'''||''''# of nodes (laki)'''||'''# of nodes (laki2)
|-
|''nehalem, mem12gb''||No longer available !||||||
|-
|''[[sb| sb, mem32gb]]''||cpu type intel sandy bridge, 32GB memory|| 2 octa core-CPU per node||98||178
|-
|''[[hsw|hsw128gb10c]]''||cpu type intel haswell, 128GB memory|| 2 x 10 core-CPU 2.6 ghz per node||84||0
|-
|''[[hsw|hsw256gb10c]]''||cpu type intel haswell, 256GB memory|| 2 x 10 core-CPU 2.6 ghz per node||4||0
|-
|''[[hsw|hsw128gb12c]]''||cpu type intel haswell, 128GB memory|| 2 x 12 core-CPU 2.5 ghz per node||344||0
|-
|''[[hsw|hsw256gb12c]]''||cpu type intel haswell, 256GB memory|| 2 x 12 core-CPU 2.5 ghz per node||16||0
|-
|''tesla''||No longer available !|| ||||
|-
|''mem64gb''||[[sb]] node with 64GB memory|| ||30||10
|-
|''mem128gb''||[[sb]] node with 128GB memory|| ||0||4 (1 shared)
|-
|''[[mem256gb|mem256gb, scratch4tb ]]''||interlagos node with 256GB memory, 4 nodes have additional a 4TB local scratch disk|| ||10||0
|-
|''mem384gb, scratch11tb,k20xm ''||node with intel IvyBridge@3.3Ghz, 16 cores, 384GB memory, 11TB local SSD scratch disk, Tesla K20Xm|| only for single node jobs available ||3||0
|-
|''smp''||cpu type intel nehalem, 1TByte memory,8 socket 6 core CPU's|| will be shared with other jobs!<font color=red> Please use <tt>"qsub -q smp -l nodes=1:smp..." </tt></font>||1
|-
|''mem144gb, scratch6tb, scratch2tb''||cpu type intel nehalem, 148GB memory, 2TB local scratch (1 nodes) or 6TB local scratch (1 nodes)|| 2 octa core-CPU per node + local scratch disk||1
|-
|''vis''||Graphic node Nvidia Quadro FX 5800, 8 core intel W3540, 24GB memory|| only 1 node per job! <font color=red> Please use <tt>"qsub -q vis -l nodes=1:vis..." </tt></font>||3||1
|-
|''gtx680''||Cuda Node with GTX680, 8 core intel E5540, 12GB memory|| only 1 node per job! ||2||0
|}
 
Multi nodes can be specified using a ''+'': <pre>nodes=2:tesla+3:nehalem</pre> The example above will allocate 2 nodes with feature ''tesla'' and 3 nodes with feature ''nehalem''.</ul>
<li><pre>walltime=<time></pre></li>
</OL>
 
==== Usage of multi-socket nodes and multi-core cpus  ====
The batch system takes into account the number of cpus and nodes (summarized PE) for each node when assigning resources.
----
'''The resource request feature ''nodes''.''' If the request has no option for the number of PE per node (ppn) set, then the system assumes that the requested nodes is equal the number of PE's. The batch system allocates a number of nodes that in total fulfill the sum of requested PE.  
    
    


'''Example:'''
'''Example:'''
   qsub -l nodes=2:nehalem ./myscript
   qsub -l select=2:node_type=skl ./myscript
The batch system allocates two node of feature nehalem. The file <tt>${PBS_NODEFILE}</tt> contains:
The batch system allocates two node of type skl. The file <tt>${PBS_NODEFILE}</tt> contains:
   node1
   node1
   node2
   node2
----
----
'''The ressource request feature ''nodes'', Option ''ppn''.''' If the request has the option <tt>ppn</tt> defined, then it is possible to allocate the defined PE's on a node. This option especially allow OpenMPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.  
If the chunk request has the option <tt>mpiprocs</tt> defined, then it is possible to allocate the defined PE's on a node. This option especially allow OpenMPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.  




'''Example:'''
'''select example with 2 chunk requests (seperated by '+'):'''
   qsub -l nodes=2:nehalem:ppn=2+1:nehalem:ppn=3 ./myscript
   qsub -l select=2:skl:mpiprocs=2+1:node_type=skl:mpiprocs=3 ./myscript
The batch system allocates 2 nodes of feature nehalem each for 2 PE's and 1 node of feature nehalem for 3 PE's. Then the file <tt>${PBS_NODEFILE}</tt> contains:
The batch system allocates 2 nodes of type skl each for 2 PE's and 1 node of type skl for 3 PE's. Then the file <tt>${PBS_NODEFILE}</tt> contains:
   node1
   node1
   node1
   node1
Line 180: Line 185:
   node3
   node3
   node3
   node3
----
'''The resource request feature ''nodes'', option ''pmem''.''' For special applications it will be useful to allocate only 1 PE per node. In this case you need a little trick, because the batch system handle the option ''ppn=1'' like the requests without this option (see the first example). Therefore you have to define an additional option in your request. A simple way to do this is to request the maximum of the node's memory.


'''Example:'''
=== Defaults for Ressource Requests ===
  qsub -l nodes=2:nehalem,pmem=11gb ./myscript
The batch system allocates 2 nodes of feature nehalem. Each of the nodes with feature nehalem have 12GByte RAM installed, so the batch system is constrained to allocate 2 nodes. And the file <tt>${PBS_NODEFILE}</tt> should look like this:
  node1
  node2
 
==== Defaults for Ressource Requests ====
If you don't set the resources for your job request, then you will get default resource limits for your job.
If you don't set the resources for your job request, then you will get default resource limits for your job.
{|border="1" cellpadding="2"
{|border="1" cellpadding="2"
|-
|-
|'''feature'''||'''value'''||'''notes'''
|'''resource'''||'''value'''||'''notes'''
|-
|-
|walltime||00:10:00|||
|select||1|||
|-
|-
|nodes||1|||
|mpiprocs||1|||
|-
|ppn||1|||
|}
|}


Please select your resource requests carefull. The higher your specified resource limits the lower the job priority.
Please select your resource requests carefull.  
See also [[NEC_Cluster_QueuePolicies_for_(laki_%2B_laki2)]].


To have the same '''environmental settings''' (exported environment) of your current session in your batchjob, the ''qsub'' command needs the option argument ''-V''.
To have the same '''environmental settings''' (exported environment) of your current session in your batchjob, the ''qsub'' command needs the option argument ''-V''.


==== Run job on other Account ID ====
=== Run job on other Account ID ===


There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the
There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the
groupname of this project has to be passed  in the group_list:
groupname of this project has to be passed  in the group_list:


<pre>qsub -l nodes=1:nehalem -W group_list=<groupname></pre>
<pre>qsub -l select=1:node_type=hsw -W group_list=<groupname></pre>


==== Usage of a Reservation ====
To get your available groups:


For nodes which are reserved for special groups or users, you need to specify an additional option for this reservation:
<pre>id -Gn</pre>
: E.g. a reservation named ''john.1'' will be used with following command:
{{Warning|text= note that this procedure is neither applicable nor necessary for the default project
   
(associated to the primary group), printed with "id -gn".}}
qsub -l nodes=1:nehalem,walltime=1:00 -W x=FLAGS:ADVRES:john.1 testjob.cmd


==== Job Arrays ====
=== Usage of a Reservation ===


Job arrays are groups of simlar jobs. Those jobs usually have slightly different parameters which depend on the current job index.
For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:
This job index will be available in the $PBS_ARRAYID variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.
: E.g. a reservation of some nodes is bound to the queue named ''workday'':
qsub -q workday -l select=1:node_type=hsw -l walltime=1:00 testjob.cmd


Job arrays can be requested with '-t <range>[%<count_of_parallel_processes>]'.
Following reservations are availalble at the moment:


The range is specifed as list of comma separated values. The values may be individual integers as well as integer ranges, spaces are not allowed. A count of parallel running processes can be requested. It must be specificed as last in the array request and is delimited from the array range by a percent sign (%). For further details see the [http://www.adaptivecomputing.com/resources/docs/torque/Content/topics/commands/qsub.htm#-t documentation].
{|border="1" cellpadding="2"
 
|-
In order to request a job array with job IDs 0,9,1000-1017 and restrict parallel running jobs to two you can use the following:
|'''reservation'''||'''resources'''||'''queue'''||'''limitations'''||'''notes'''
<pre> qsub -t 1,9,1000-1017%2 myJobScript</pre>
|-
or inside the job script
|S150017||1 node of node_type hsw (20 cores, 128GB)||workday for routing to S150017||valid on Mo-Fr 6:00-18:00, user- and group-run limits, walltime limit, nodecount limits will be set for jobs using this reservation||This daytime reservation should be used for jobs with a short walltime
<pre>
|}
#!/bin/bash
#PBS -l nodes=1,walltime=00:01:00
#PBS -t 1,9,1000-1017%2
 
# Replace the following line with your own code
echo "This is job with id '$PBS_ARRAYID' running on compute node '`hostname`'";
exit 0;
</pre>
 
In case the job id returned by qsub was '23' then the job array will result in showq-output similar to the following:
<pre>
active jobs------------------------
JOBID              USERNAME STATE PROCS  REMAINING            STARTTIME
 
23[1]              testuser    Running    1    00:00:57  Thu Dec 20 17:38:10
23[9]              testuser    Running    1    00:00:57  Thu Dec 20 17:38:10
 
2 active jobs               2 of 3 processors in use by local jobs (66.67%)


eligible jobs----------------------
=== Job Arrays ===
JOBID              USERNAME STATE PROCS    WCLIMIT            QUEUETIME


Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index.
This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.


0 eligible jobs
Job arrays can be requested with
<pre>qsub -J <range> <my_array_jobscript></pre>


blocked jobs-----------------------
range is specified in the form X-Y[:Z]  where X is the first index, Y is the upper bound on the indices and Z is the stepping factor.  For example,  2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.
JOBID              USERNAME STATE PROCS    WCLIMIT            QUEUETIME


23[1000]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
{{Note|text =
23[1001]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
<li> Job arrays cannot be interactive</li>
23[1002]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
<li> Job arrays are automatically marked as rerunnable</li>
23[1003]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
}}
23[1004]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
..
23[1017]          testuser   Hold    1    00:01:00  Thu Dec 20 17:35:39
</pre>


=== Examples ===
=== Examples ===


==== Examples for PBS options in job scripts ====
==== Examples for PBS options in job scripts ====
You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBS (Torque) directives (#PBS ...) for the options of qsub looks like this:
<UL>
<li>
You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBSPro directives (#PBS ...) for the options of qsub looks like this:
<pre>
<pre>
#!/bin/bash
#!/bin/bash
#
#
# Simple PBS batch script that reserves two cpus and runs one
# Simple PBS batch script that reserves two exclusive Skylake nodes
# MPI process on each node)
# and runs only one MPI process on each node (in total 2 MPI processes)
# The default walltime is 10min !
# The walltime is 10min
#
#
#PBS -l nodes=2:nehalem
#PBS -l select=2:node_type=skl:mpiprocs=1
cd $HOME/testdir
#PBS -l walltime=00:10:00
mpirun -np 2 -hostfile $PBS_NODEFILE ./mpitest
 
</pre>
### go to directory where your job request was submitted
cd $PBS_O_WORKDIR


'''VERY important''' is that you specify a shell in the first line of your batch script.
### load for example the Intel MPI environment
module load mpi/impi


'''VERY important''' in case you use the openmpi module is to omit the -hostfile option. Otherwise an error will occur like
### run you parallel application on the allocated nodes
<pre>
mpirun -np 2 ./mpitest
[n110402:02618] pls:tm: failed to poll for a spawned proc, return status = 17002
[n110402:02618] [0,0,0] ORTE_ERROR_LOG: In errno in file ../../../../../orte/mca/rmgr/urm/rmgr_urm.c at line 462
[n110402:02618] mpirun: spawn failed with errno=-11
</pre>
</pre>


{{Warning|text= <BR>
# you have to specify a shell in the first line of your batch script
# you have to specify the number of nodes you need and the node type
# you have to specify the walltime the job needs
# allocated nodes will not be shared with other jobs, even though you uses the nodes only partial
}}
</li>


If you want to use two MPI processes on each node this can be done like this:
<BR>
<li>
If you want to use four MPI processes on each node this can be done like this:
<pre>
<pre>
#!/bin/bash
#!/bin/bash
#
#
# Simple PBS batch script that reserves two nodes and runs a
# Simple PBS batch script that reserves two exclusive Skylake nodes
# MPI program on four processors (two on each node)
and runs four MPI process on each node (in total 8 MPI processes)
# The default walltime is 10min !
# The walltime is 10min
#
#
#PBS -l nodes=2:nehalem:ppn=2
#PBS -l select=2:node_type=skl:mpiprocs=4
cd $HOME/testdir
#PBS -l walltime=00:10:00
mpirun -np 4 -hostfile machines ./mpitest
 
### go to directory where your job request was submitted
cd $PBS_O_WORKDIR
 
### load for example the Intel MPI environment
module load mpi/impi
 
### run you parallel application on the allocated nodes
mpirun -np 8 ./mpitest
</pre>
</pre>
</li>


<BR>
<li>
If you need 2h wall time and one node you can use the following script:
If you need 2h wall time and one node you can use the following script:
<pre>
<pre>
#!/bin/bash
#
#
# Simple PBS batch script that runs a scalar job using 2h
# Simple PBS batch script that runs a scalar job  
# on 1 Skylake node using 2h
#
#
#PBS -l nodes=1:nehalem,walltime=2:00:00
#PBS -l select=1:node_type=skl,walltime=2:00:00
cd $HOME/jobdir
cd $PBS_O_WORKDIR
./my_executable
./my_executable
</pre>
</pre>
</li>
</UL>


==== Examples for starting batch jobs: ====
==== Examples for starting batch jobs: ====
Line 327: Line 325:
<pre>qsub <script></pre>
<pre>qsub <script></pre>
</li>
</li>
<BR>
<LI>
<LI>
Starting a script using 3 nodes and a real time of 2 hours:
Starting a script using 3 nodes of node_type 'skl' and a real time of 2 hours:
<pre>qsub -l nodes=3:nehalem,walltime=2:00:00 <script></pre>
<pre>qsub -l select=3:node_type=skl,walltime=2:00:00 <script></pre>
</li>
</li>
<BR>
<li>
<li>
Starting a script using 5 cluster nodes using 4 processors on each node with PBS Feature nehalem:
Starting a script using 5 cluster nodes of node_type 'hsw' using 4 processors on each node:
<pre>qsub -l nodes=5:nehalem:ppn=4,walltime=2:00:00 <script></pre>
<pre>qsub -l select=5:node_type=hsw:mpiprocs=4,walltime=2:00:00 <script></pre>
</li>
</li>
<BR>
<li>
<li>
Starting a script using 1 cluster node with PBS Feature mem24gb and 5 processors with PBS Feature mem12gb and real job time of 1.5 hours:
Starting a script using 1 cluster node of type 'hsw' with 256GB memory and additional requesting 5 other 'hsw' nodes (24 cores) with 128GB memory and using 24 MPI processes on each of this 5 nodes;  real job time is 1.5 hours:
<pre>qsub -l nodes=1:mem24gb+5:mem12gb,walltime=1:30:00 <script></pre>
<pre>qsub -l select=1:node_type=hsw:node_type_mem=256gb+5:node_type=hsw:node_type_core=24c:node_type_mem=128gb:mpiprocs=24,walltime=1:30:00 <script></pre>
{{Warning|text =
Because in the first chunk (''1:node_type=hsw:node_type_mem=256gb'') the number of cores for the node isn't specified, it's possible you will get a 20 core hsw node or a 24 core hsw node with 256 GB.
The second chunk is clearly specified and you will get 5 hsw nodes each will have 128 GB and 20 cores.
}}
</li>
</li>
<BR>
<li>
<li>
Starting a interactive batch job using 5 processors with a job real time of 300 seconds:
Starting a interactive batch job using 5 hsw nodes regardless how much memory or cores and with a job real time of 300 seconds:
<pre>qsub -I -l nodes=5:nehalem,walltime=300</pre>
<pre>qsub -I -l select=5:node_type=hsw,walltime=300</pre>
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command cat $PBS_NODEFILE on the batch job shell or with the PBS status command qstat -n on the master node.  
{{Note|text =
You can log in from the frontend or any assigned node to all other assigned nodes by <pre>ssh <nodename></pre>
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command  
<pre>cat $PBS_NODEFILE </pre>
on the batch job shell or with the PBS status command  
<pre>qstat -n <jobid></pre> on the master node.  
You can log in from the frontend or any assigned node to all other assigned nodes by  
<pre>ssh <nodename></pre>
If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.
If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.
}}
</li>
</li>
<BR>
<li>
<li>
Possibilities to request 4 cluster nodes (with feature 'nehalem' i.e. 32 processors).
Other possibilities to request cluster nodes. You want 4 nodes with a minimum of 20 cores and 128 GB memory each regardless which type of nodes. And you want to run 20 MPI processes on each of the 4 nodes:
<pre>qsub -l nodes=4:nehalem:ppn=8 <script></pre>
<pre>qsub -l select=4:ncpus=20:mem=128g:mpiprocs=20 <script></pre>
<pre>qsub -l nodes=4:nehalem:pmem=11gb <script></pre>
Then its possible you will get a mix of 4 nodes composed of hsw node types with different memory size or core size.
The difference between both job submisson kinds is in the content of PBS_NODEFILE.
 
Another possibilty to get explicit hsw nodes:
<pre>qsub -l select=4:node_type=hsw:mem=256g:ncpus=20 <script></pre>
Because the qsub argument 'ncpus=20' requests nodes with at least 20 cores each, its possible you will get available hsw nodes with 20 cores or with 24 cores and with at least 256 GB memory.
</li>
</li>
<BR>
<li>
<li>
Starting a script which should run on other Account ID:  
Starting a script which should run on other Account ID:  
Line 357: Line 380:
<pre>id</pre>
<pre>id</pre>
Choose a valid groupname for your job (abc12345 will serve as a placeholder here):
Choose a valid groupname for your job (abc12345 will serve as a placeholder here):
<pre>qsub -l nodes=5:nehalem,walltime=300 -W group_list=abc12345</pre>
<pre>qsub -l select=5:node_type=hsw,walltime=300 -W group_list=abc12345</pre>
</li>
</li>
</UL>
</UL>


== Get the batch job status ==
== Get the batch job status ==
<font color=red>Please, don't run the commands '''qstat''' or '''showq''' in a iteration loop using e.g. '''watch'''.
Those commands will block all other users. You can use '''nstat''' in place of '''qstat''' or '''showq'''. This command uses a SQL database which will be updated every 3 minutes. </font>
<li>
<li>
'''Available commands'''
'''Available commands'''
Line 369: Line 390:
<pre>
<pre>
qstat [options]
qstat [options]
showq [options] (showq -h for details)
nstat
nstat
batchstat [options]
</pre>
</pre>
For detailed informations, see man pages:
For detailed informations, see man pages:
<pre>
<pre>
nstat -h
nstat -h
batchstat -h
man qstat
man qstat
man pbsnodes
man pbsnodes
Line 383: Line 405:
'''Examples'''
'''Examples'''
   <ul>
   <ul>
list all batch jobs:
list all your own batch jobs:
<font color=red>(query status of jobs owned by other users has been prohibited!)</font>
<pre>qstat -a</pre>
<pre>qstat -a</pre>
list all batch jobs (anonymous)
<pre>batchstat</pre>
lists all batch queues with resource limit settings:
lists all batch queues with resource limit settings:
<pre>qstat -q</pre>
<pre>qstat -q</pre>
lists node information of a batch job ID:
lists node information of one of your batch job ID:
<pre>qstat -n <JOB_ID></pre>
<pre>qstat -n <JOB_ID></pre>
lists detailed information of a batch job ID:
lists detailed information of one of your batch job ID:
<pre>qstat -f <JOB_ID></pre>
<pre>qstat -f <JOB_ID></pre>
Displays estimated start time for your queued jobs
<pre>qstat -T <JOB_ID></pre>
Displays status information for your jobs, job arrays, and subjobs:
<pre>qstat -t <JOB_ID></pre>
lists information of the PBS node status:
lists information of the PBS node status:
<pre>
<pre>
Line 399: Line 428:
<pre>
<pre>
nstat
nstat
batchstat
</pre>
</pre>


Line 407: Line 437:


<pre>
<pre>
frontend> qsub -l nodes=2:nehalem,walltime=300 -X -I
frontend> qsub -q vis -l select=1:node_type=fx5800,walltime=300 -X -I
</pre>
</pre>
{{Note|text=
<li>DISPLAY variable in submission environment must be set to desired display.</li>
<li>Can be used with interactive jobs only: must be used with -I </li>
<li>Cannot be used with -v DISPLAY</li>
}}
== Container, singularity, AI ==
Solving problems using artificial intelligence, usually you need nodes equipped with fast local disks and GPU's.
Some nodes (node_type=clx-ai) are able to run singularity containers.
To be able to start singularity container, the nodes have to be prepared before your batch job starts.
You can do this by a resource request ('UNS=True') at your batch submission:
  qsub -l select=1:node_type=clx-ai:UNS=True,walltime=300 <mybatchjob>
Note: Only the nodes with node_type=clx-ai are able to run singularity container!
More about singularity container software can be found here: [[Urika CS|Cray Urika CS software stack ]]
== NEC vector CPU's Aurora ==
To use the NEC vector CPU's you need to submit your jobs to a special Queue "vector":
  qsub -q vector -l select=1:node_type=aurora,walltime=300 <mybatchjob>
More Details about NEC Auora can be found [[Aurora_Tsubasa| here (Aurora Tsubasa Platform)]].
== Queue Policies and Limitations ==
Different job queues are available for efficient resource usage.
In most cases users do not need to declare a job queue with the qsub command. Jobs are sorted to the right class automatically.
In the following the definition for each job queue is given. In general jobs
with a Duration up to <font color=red>'''24 hours''' </font> and half of the available resources can be
submitted.
Some special resources like nodes with graphics or very large memory, (job sharing nodes) are only available for special queues which have to be declared with the qsub command.
For larger jobs or for special job requirements different restrictions are in place respectively you have to consult the project team
Each limit settings and policies could be changed in future to adjust the cluster usage for new user requirements.
At the moment following queues and policies are defined:
=== route (default) ===
If users don't declare a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs (academic users) are: 
<UL>
==== single ====
This queue is available for all single node jobs.
{|border="1" cellpadding="2"
|-
|'''resource'''||'''min'''||'''max''' || '''note'''
|-
|walltime|| || 24 hours ||
|-
|available nodes||  || 1 per job (64 in total) || only single node jobs
|-
|priority|| || || low
|-
|joblimit|| || 20 per user, 30 per group ||
|}
==== normal ====
This is for all regular parallel jobs using 2 nodes and more.
{|border="1" cellpadding="2"
|-
|'''resource'''||'''min'''||'''max''' || '''note'''
|-
|walltime|| || 24 hours ||
|-
|available nodes|| 2 || 384 per job (425 in total) ||
|-
|priority|| || || normal
|-
|joblimit|| || 20 per user ||
|}
</UL>
=== test ===
This queue is for tests and development with restricted resources needs. The jobs in this queue are expected to deliver results after very short time. It's forbidden to use this queue for production jobs. Users have to declare this queue with the qsub submission.
{|border="1" cellpadding="2"
|-
|'''resource'''||'''min'''||'''max''' || '''note'''
|-
|walltime|| || 25 minutes ||
|-
|available nodes|| || 40 ||
|-
|priority|| || || very high
|-
|joblimit|| || 1 per user (4 for ALL) ||
|}
=== interactive ===
This queue is also only for batch jobs in [[Batch_System_PBSPro_(laki_%2B_laki2)#Interactive_batch_Mode | interactive batch mode]] which can also be for tests and development. Users can not declare this queue with the qsub submission. But all [[Batch_System_PBSPro_(laki_%2B_laki2)#Interactive_batch_Mode | interactive batch jobs]] will be routed to this queue automatically.
{|border="1" cellpadding="2"
|-
|'''resource'''||'''min'''||'''max''' || '''note'''
|-
|walltime|| || 8 hours ||
|-
|available nodes|| || 32 || only for job in interactive batch mode
|-
|priority|| || || very high
|-
|joblimit|| || 2 per user ||
|}
=== vis ===
This queue is only available for jobs using graphic nodes. It can also be used in interactive batch mode.
Only the nodes with a graphic card installed are available with this queue. Users have to declare this queue with the qsub job submission.
{|border="1" cellpadding="2"
|-
|'''resource'''||'''min'''||'''max''' || '''note'''
|-
|walltime|| || 10 hours ||
|-
|available nodes|| || 1 per job (4 in total) || only nodes with graphic cards
|-
|priority|| || || high
|-
|joblimit|| || 2 per user, 4 per group ||
|}
=== Job Run Limitations ===
*'''The maximum time limit for a Job is 24hours.'''
* User limits:
** limited number of jobs of one user that can run at the same time
** in total a user can only allocate 384 nodes.
* User Group limits:
** limited number of jobs of users in the same group that can run at the same time
* Batch Queue limits of all user jobs:
** not all nodes / node types are available on each queue (visualisation nodes can not be used in multi node job queues)
** The number of jobs for each user in the different job queues are restricted. If you reach this number you can submit further jobs when prior jobs have ended.
*** (If more jobs are submitted than allowed for one job queue the old ones will be placed in the dispatcher queue 'route' and will move up in the proper destination queue after jobs from this user in the corresponding queue have ended. The waiting queue for each user will take up to 10 jobs. With this it is possible to submit job ahead.)
=== Queues with extended wall time limits ===
are not available in general. This Queue '''spec1''' is available for Jobs, which can not run within the 24h timeframe. Access to this queue is only granted by passing an evaluation process. Following rules apply to this queue:
* Jobs may be killed for operation reasons at any time.
* Jobs will be accounted in any case. This is also true if the job has to be terminated for operational reasons.
* Joblimit per Group = 1
* Joblimit per user = 1
* Total number of nodes used for this queue = 64
* Only general compute node types available, no visualisation or special node types
* Low scheduling priority
* Max walltime  96h

Revision as of 09:32, 17 March 2021

Introduction

Applications cannot be run directly from the login node on the compute nodes of the system. The only way to get access to the compute nodes of the system is to use the batch system. Therefore a batch script has to be provided, which includes all the necessary commands to run the application. The installed batch system is PBSPro

You generally interact with the batch system in two ways: (1) through options specified in job submission scripts (these are detailed below in the examples) and (2) by using PBSPro commands on the login nodes.

There are three key commands used to interact with PBSPro:

qsub
used to submit your batch script to the system to be executed as a batch job
qstat
to display information about submitted batch jobs
qdel
to discard a submitted or running batch job

Check the man pages of the individual commands as well as the man page for PBSPro on the login node for more information:

man qsub

man qstat

man qdel

man pbs_professional


If you are looking for in depth information consult the User Guide in the PBS Pro documentation.


Requesting Resources using batch system

Resources are allocated to jobs both by explicitly requesting them and by applying specified defaults.
Jobs explicitly request resources either at the host level in chunks defined in a selection statement, or in job-wide resource requests.

    Format:
  • job wide request:
       qsub ... -l <resource name>=<value> 

    The only resources that can be in a job-wide request are server-level or queue-level resources, such as walltime.

  • selection statement:
       qsub ... -l select=<chunks> 

    The only resources that can be requested in chunks are host-level resources, such as mem and ncpus. A chunk is the smallest set of resources that will be allocated to a job. It is one or more resource_name=value statements separated by a colon, e.g.:

    ncpus=2:mem=32GB
    A  selection statement is of the form:
    
      -l select=[N:]chunk[+[N:]chunk ...] 
    Note: If N is not specified, it is taken to be 1. No spaces are allowed between chunks.


Warning: all requested cluster nodes will be exclusively allocated by 1 job. The default nodes can not be shared by multiple jobs. The allocated nodes of your job will be accounted completely, even though your job uses the allocated nodes only partial

Node types

You have to specify the resources you need for your batch job. These resources are specified by including them in the -l argument (selection statement and job-wide resources) on the qsub command or in the PBS job script. The 2 important resources you have to specify are number of nodes of a specific node type in the selection statement and the walltime in the job-wide resource request you need for this job:

  1. select=<number of nodes>:<node_resource_variable=type>
    • To distinguish between different nodes 4 node resource variables are assigned to each node. The node_type, node_type_cpu, node_type_mem and node_type_core of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes. For some special nodes there are also the resources localscratch and linkspeed available.
      Available node types:
      node_type node_type_cpu node_type_mem node_type_core Graphic / Accelerator localscratch linkspeed describtion notes # of nodes (vulcan) # of nodes (vulcan2)
      clx-21 CascadeLake@2.10GHz 384gb 40c 1.8TB HDR-100 Intel Xeon Gold 6230 @ 2.5GHz, CascadeLake, 384GB memory, 1.8TB /localscratch 2 x 20 core-CPU per node 8 0
      clx-25 CascadeLake@2.50GHz 384gb 40c HDR-100 Intel Xeon Gold 6248 @ 2.5GHz, CascadeLake, 384GB memory 2 x 20 core-CPU per node 84 0
      hsw Haswell@2.60GHz 128gb 20c QDR Intel Xeon E5-2660v3 @ 2.60GHz, Haswell, 128GB memory 2 x 10 core-CPU per node 76 0
      hsw Haswell@2.60GHz 256gb 20c QDR Intel Xeon E5-2660v3 @ 2.60GHz, Haswell, 256GB memory 2 x 10 core-CPU per node 4 0
      hsw Haswell@2.50GHz 128gb 24c QDR Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 128GB memory 2 x 12 core-CPU per node 152 192 (1 is in shared mode)
      hsw Haswell@2.50GHz 128gb 24c FDR Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 128GB memory 2 x 12 core-CPU per node 0 0
      hsw Haswell@2.50GHz 256gb 24c QDR Intel Xeon E5-2680v3 @ 2.50GHz, Haswell, 256GB memory 2 x 12 core-CPU per node 16 0
      skl Skylake@2.0GHz 192gb 40c EDR Intel Xeon Gold 6138 @ 2.00GHz, Skylake, 92GB memory 2 x 20 core-CPU per node 88 0
      smp Skylake@2.40GHz 1536gb 40c FDR Intel Gold 6148 @ 2.40GHz, Skylake, 1.5TByte memory 2 x 20 core-CPU per node
      will be shared with other jobs! Please use "qsub -q smp -l select=1:node_type=smp ..."
      1 (shared) 0
      aurora Skylake@2.60GHz 192gb 24c 8x NEC Aurora vector CPUs EDR 2.60GHz, Skylake, 192GB memory, 8 NEC Aurora TSUBASA nodes 2 x 12 core-CPU per node
      Please use "qsub -q vector -l select=1:node_type=aurora ..."
      8 0
      clx-ai CascadeLake@2.60GHz 768gb 36c 8x Tesla V100 SXM2 32GB 7.3TB 4x HDR-100 Intel Xeon Gold 6240 @ 2.6GHz, CascadeLake, 768GB memory, 7.3TB /localscratch, 220GB /tmp on SSD, 8x Tesla V100 SXM2 32GB 2 x 18 core-CPU + 8 GPU's per node 1 0
      k20xm IvyBridge@3.3Ghz 384gb 16c Nvidia Tesla K20Xm 11TB QDR Intel E5-2667v2 @ 3.30GHz, IvyBridge, 384GB memory, 11TB local SSD scratch disk, Nvidia Tesla K20Xm 2 x 8 core-CPU per node
      only for single node jobs available
      3 0
      p100 Broadwell@3.00GHz 1024gb 24c Nvidia Tesla P100 11TB QDR Cuda Node with Nvidia Tesla P100
      Intel Xeon E5-2687Wv4 @ 3.0GHz, Broadwell, 1TB memory, 11TB local SSD scratch disk
      2 x 8 core-CPU per node
      reserved for special users! Please use "qsub -q R_i1 -l select=1:node_type=p100 ..."
      3 0
      visamd Skylake@2.60GHz 96gb 8c AMD Radeon PRO WX 8200 HDR-100 Graphic Node with AMD Radeon PRO WX 8200
      Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory
      2 x 4 core-CPU per node
      only 1 node per job! Please use "qsub -q vis -l select=1:node_type=visamd ..."
      6 0
      visnv Skylake@2.60GHz 96gb 8c Nvidia Quadro RTX 4000 HDR-100 Graphic Node with Nvidia Quadro RTX 4000
      Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory
      2 x 6 core-CPU per node
      only 1 node per job! (1 node is reserved for special users) Please use "qsub -q vis -l select=1:node_type=visnv ..."
      2 1
      visp100 Broadwell@3.20GHz 256gb 16c Nvidia Tesla P100 FDR Cuda Node with Nvidia Tesla P100
      Intel Xeon E5-2667v4 @ 3.2GHz, Broadwell, 256GB memory
      2 x 8 core-CPU per node
      only 1 node per job! Please use "qsub -q vis -l select=1:node_type=visp100 ..."
      10 0


      Multi node type job can also be specified using a +:

        select=1:node_type=hsw:node_type_mem=256gb+3:node_type=hsw:node_type_mem=128gb:node_type_core=20c

        The example above will allocate 1 hsw node (a 20 core ore 24 core type) with 256 GB memory and 3 hsw nodes (the 20 cores type) with 128 GB memory.


      To allocate special nodes with local disk you can use the special node resource variable localscratch:

        select=1:node_type=clx-21:localscratch=1800GB

      Or you want hsw nodes with network interconnect QDR, then you can use the special node resource variable linkspeed:

        select=4:node_type=hsw:linkspeed=QDR
  2. walltime=<time>

Batch Mode

Production jobs are typically run in batch mode. Batch scripts are shell scripts containing flags and commands to be interpreted by a shell and are used to run a set of commands in sequence.

  • The number of required nodes, cores, wall time and more can be determined by the parameters in the job script header with "#PBS" before any executable commands in the script.
#!/bin/bash
#PBS -N job_name
#PBS -l select=2:node_type=hsw:mpiprocs=24
#PBS -l walltime=00:20:00             
  
# Change to the direcotry that the job was submitted from
cd $PBS_O_WORKDIR

# using the INTEL MPI module
module load mpi/impi

# Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes
mpirun -np 48  ./my_mpi_executable arg1 arg2 > my_output_file 2>&1
  • The job is submitted by the qsub command (all script head parameters #PBS can also be adjusted directly by qsub command options).
 qsub my_batchjob_script.pbs
  • Setting qsub options on the command line will overwrite the settings given in the batch script:
 qsub -N other_name -l select=2:node_type=hsw:mpiprocs=24 -l walltime=00:20:00 my_batchjob_script.pbs
  • The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.
  • At the end of the execution output and error files are returned to your HOME directory
  • This example will run your executable "my_mpi_executable" in parallel with 48 MPI processes (mpiprocs=24 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 24 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The mpirun example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 48 MPI processes with 24 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.
Note:
  • While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the
    qsub -koed my_batchjob_script.pbs
    option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.

  • Interactive batch Mode

    Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:

     qsub -I -l select=2:node_type=hsw:ncpus=24:mpiprocs=24 -l walltime=00:30:00
    

    If the requested resources are available and free (in the example above: 2 hsw nodes/24 cores each, 30 minutes, prepared for 24 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources. Now you have to use the mpirun command to launch your parallel application to the allocated compute nodes. When you are finished, enter logout to exit the batch system and return to the normal command line.


    PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus)

    In most MPI environments, the PBS_NODEFILE will be usefull to start the correct number of mpi processes on each allocated node. The jobs ${PBS_NODEFILE} contents depends on the number of MPI processes for each requested chunk. Inside a select statement of each chunk you can define a mpiprocs option (Type: integer). The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.


    Example:

     qsub -l select=2:node_type=skl ./myscript
    

    The batch system allocates two node of type skl. The file ${PBS_NODEFILE} contains:

     node1
     node2
    

    If the chunk request has the option mpiprocs defined, then it is possible to allocate the defined PE's on a node. This option especially allow OpenMPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.


    select example with 2 chunk requests (seperated by '+'):

     qsub -l select=2:skl:mpiprocs=2+1:node_type=skl:mpiprocs=3 ./myscript
    

    The batch system allocates 2 nodes of type skl each for 2 PE's and 1 node of type skl for 3 PE's. Then the file ${PBS_NODEFILE} contains:

     node1
     node1
     node2
     node2
     node3
     node3
     node3
    

    Defaults for Ressource Requests

    If you don't set the resources for your job request, then you will get default resource limits for your job.

    resource value notes
    select 1
    mpiprocs 1

    Please select your resource requests carefull.

    To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.

    Run job on other Account ID

    There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the groupname of this project has to be passed in the group_list:

    qsub -l select=1:node_type=hsw -W group_list=<groupname>

    To get your available groups:

    id -Gn
    Warning: note that this procedure is neither applicable nor necessary for the default project (associated to the primary group), printed with "id -gn".


    Usage of a Reservation

    For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:

    E.g. a reservation of some nodes is bound to the queue named workday:
    qsub -q workday -l select=1:node_type=hsw -l walltime=1:00 testjob.cmd
    

    Following reservations are availalble at the moment:

    reservation resources queue limitations notes
    S150017 1 node of node_type hsw (20 cores, 128GB) workday for routing to S150017 valid on Mo-Fr 6:00-18:00, user- and group-run limits, walltime limit, nodecount limits will be set for jobs using this reservation This daytime reservation should be used for jobs with a short walltime

    Job Arrays

    Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index. This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.

    Job arrays can be requested with

    qsub -J <range> <my_array_jobscript>

    range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.

    Note:
  • Job arrays cannot be interactive
  • Job arrays are automatically marked as rerunnable

  • Examples

    Examples for PBS options in job scripts

    • You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBSPro directives (#PBS ...) for the options of qsub looks like this:
      #!/bin/bash
      #
      # Simple PBS batch script that reserves two exclusive Skylake nodes
      # and runs only one MPI process on each node (in total 2 MPI processes)
      # The walltime is 10min
      #
      #PBS -l select=2:node_type=skl:mpiprocs=1
      #PBS -l walltime=00:10:00
      
      ### go to directory where your job request was submitted 
      cd $PBS_O_WORKDIR
      
      ### load for example the Intel MPI environment
      module load mpi/impi
      
      ### run you parallel application on the allocated nodes
      mpirun -np 2 ./mpitest
      
      Warning:
      1. you have to specify a shell in the first line of your batch script
      2. you have to specify the number of nodes you need and the node type
      3. you have to specify the walltime the job needs
      4. allocated nodes will not be shared with other jobs, even though you uses the nodes only partial

    • If you want to use four MPI processes on each node this can be done like this:
      #!/bin/bash
      #
      # Simple PBS batch script that reserves two exclusive Skylake nodes
      #  and runs four MPI process on each node  (in total 8 MPI processes)
      # The walltime is 10min
      #
      #PBS -l select=2:node_type=skl:mpiprocs=4
      #PBS -l walltime=00:10:00
      
      ### go to directory where your job request was submitted
      cd $PBS_O_WORKDIR
      
      ### load for example the Intel MPI environment
      module load mpi/impi
      
      ### run you parallel application on the allocated nodes
      mpirun -np 8 ./mpitest
      

    • If you need 2h wall time and one node you can use the following script:
      #!/bin/bash
      #
      # Simple PBS batch script that runs a scalar job 
      # on 1 Skylake node using 2h
      #
      #PBS -l select=1:node_type=skl,walltime=2:00:00
      cd $PBS_O_WORKDIR
      ./my_executable
      

    Examples for starting batch jobs:

    • Starting a script with all options specified inside the script file
      qsub <script>

    • Starting a script using 3 nodes of node_type 'skl' and a real time of 2 hours:
      qsub -l select=3:node_type=skl,walltime=2:00:00 <script>

    • Starting a script using 5 cluster nodes of node_type 'hsw' using 4 processors on each node:
      qsub -l select=5:node_type=hsw:mpiprocs=4,walltime=2:00:00 <script>

    • Starting a script using 1 cluster node of type 'hsw' with 256GB memory and additional requesting 5 other 'hsw' nodes (24 cores) with 128GB memory and using 24 MPI processes on each of this 5 nodes; real job time is 1.5 hours:
      qsub -l select=1:node_type=hsw:node_type_mem=256gb+5:node_type=hsw:node_type_core=24c:node_type_mem=128gb:mpiprocs=24,walltime=1:30:00 <script>
      Warning: Because in the first chunk (1:node_type=hsw:node_type_mem=256gb) the number of cores for the node isn't specified, it's possible you will get a 20 core hsw node or a 24 core hsw node with 256 GB. The second chunk is clearly specified and you will get 5 hsw nodes each will have 128 GB and 20 cores.

    • Starting a interactive batch job using 5 hsw nodes regardless how much memory or cores and with a job real time of 300 seconds:
      qsub -I -l select=5:node_type=hsw,walltime=300
      Note: For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command
      cat $PBS_NODEFILE 

      on the batch job shell or with the PBS status command

      qstat -n <jobid>
      on the master node.

      You can log in from the frontend or any assigned node to all other assigned nodes by

      ssh <nodename>
      If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated.

    • Other possibilities to request cluster nodes. You want 4 nodes with a minimum of 20 cores and 128 GB memory each regardless which type of nodes. And you want to run 20 MPI processes on each of the 4 nodes:
      qsub -l select=4:ncpus=20:mem=128g:mpiprocs=20 <script>

      Then its possible you will get a mix of 4 nodes composed of hsw node types with different memory size or core size.

      Another possibilty to get explicit hsw nodes:

      qsub -l select=4:node_type=hsw:mem=256g:ncpus=20 <script>

      Because the qsub argument 'ncpus=20' requests nodes with at least 20 cores each, its possible you will get available hsw nodes with 20 cores or with 24 cores and with at least 256 GB memory.


    • Starting a script which should run on other Account ID: First you have to know which Account ID's (groupnames) are valid for your login:
      id

      Choose a valid groupname for your job (abc12345 will serve as a placeholder here):

      qsub -l select=5:node_type=hsw,walltime=300 -W group_list=abc12345

    Get the batch job status

  • Available commands
      qstat [options]
      nstat
      batchstat [options]
      

      For detailed informations, see man pages:

      nstat -h
      batchstat -h
      man qstat
      man pbsnodes
      
  • Examples
      list all your own batch jobs: (query status of jobs owned by other users has been prohibited!)
      qstat -a

      list all batch jobs (anonymous)

      batchstat

      lists all batch queues with resource limit settings:

      qstat -q

      lists node information of one of your batch job ID:

      qstat -n <JOB_ID>

      lists detailed information of one of your batch job ID:

      qstat -f <JOB_ID>

      Displays estimated start time for your queued jobs

      qstat -T <JOB_ID>

      Displays status information for your jobs, job arrays, and subjobs:

      qstat -t <JOB_ID>

      lists information of the PBS node status:

      pbsnodes -a
      pbsnodes -l
      

      gives informatioin of PBS node and job status:

      nstat
      batchstat
      

      DISPLAY: X11 applications on interactive batch jobs

      For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.

      frontend> qsub -q vis -l select=1:node_type=fx5800,walltime=300 -X -I
      
      Note:
    • DISPLAY variable in submission environment must be set to desired display.
    • Can be used with interactive jobs only: must be used with -I
    • Cannot be used with -v DISPLAY

    • Container, singularity, AI

      Solving problems using artificial intelligence, usually you need nodes equipped with fast local disks and GPU's. Some nodes (node_type=clx-ai) are able to run singularity containers. To be able to start singularity container, the nodes have to be prepared before your batch job starts. You can do this by a resource request ('UNS=True') at your batch submission:

       qsub -l select=1:node_type=clx-ai:UNS=True,walltime=300 <mybatchjob>
      

      Note: Only the nodes with node_type=clx-ai are able to run singularity container!

      More about singularity container software can be found here: Cray Urika CS software stack

      NEC vector CPU's Aurora

      To use the NEC vector CPU's you need to submit your jobs to a special Queue "vector":

       qsub -q vector -l select=1:node_type=aurora,walltime=300 <mybatchjob>
      

      More Details about NEC Auora can be found here (Aurora Tsubasa Platform).

      Queue Policies and Limitations

      Different job queues are available for efficient resource usage.

      In most cases users do not need to declare a job queue with the qsub command. Jobs are sorted to the right class automatically. In the following the definition for each job queue is given. In general jobs with a Duration up to 24 hours and half of the available resources can be submitted. Some special resources like nodes with graphics or very large memory, (job sharing nodes) are only available for special queues which have to be declared with the qsub command. For larger jobs or for special job requirements different restrictions are in place respectively you have to consult the project team


      Each limit settings and policies could be changed in future to adjust the cluster usage for new user requirements.

      At the moment following queues and policies are defined:

      route (default)

      If users don't declare a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs (academic users) are:

        single

        This queue is available for all single node jobs.

        resource min max note
        walltime 24 hours
        available nodes 1 per job (64 in total) only single node jobs
        priority low
        joblimit 20 per user, 30 per group

        normal

        This is for all regular parallel jobs using 2 nodes and more.

        resource min max note
        walltime 24 hours
        available nodes 2 384 per job (425 in total)
        priority normal
        joblimit 20 per user

      test

      This queue is for tests and development with restricted resources needs. The jobs in this queue are expected to deliver results after very short time. It's forbidden to use this queue for production jobs. Users have to declare this queue with the qsub submission.

      resource min max note
      walltime 25 minutes
      available nodes 40
      priority very high
      joblimit 1 per user (4 for ALL)

      interactive

      This queue is also only for batch jobs in interactive batch mode which can also be for tests and development. Users can not declare this queue with the qsub submission. But all interactive batch jobs will be routed to this queue automatically.

      resource min max note
      walltime 8 hours
      available nodes 32 only for job in interactive batch mode
      priority very high
      joblimit 2 per user

      vis

      This queue is only available for jobs using graphic nodes. It can also be used in interactive batch mode. Only the nodes with a graphic card installed are available with this queue. Users have to declare this queue with the qsub job submission.

      resource min max note
      walltime 10 hours
      available nodes 1 per job (4 in total) only nodes with graphic cards
      priority high
      joblimit 2 per user, 4 per group



      Job Run Limitations

      • The maximum time limit for a Job is 24hours.
      • User limits:
        • limited number of jobs of one user that can run at the same time
        • in total a user can only allocate 384 nodes.
      • User Group limits:
        • limited number of jobs of users in the same group that can run at the same time
      • Batch Queue limits of all user jobs:
        • not all nodes / node types are available on each queue (visualisation nodes can not be used in multi node job queues)
        • The number of jobs for each user in the different job queues are restricted. If you reach this number you can submit further jobs when prior jobs have ended.
          • (If more jobs are submitted than allowed for one job queue the old ones will be placed in the dispatcher queue 'route' and will move up in the proper destination queue after jobs from this user in the corresponding queue have ended. The waiting queue for each user will take up to 10 jobs. With this it is possible to submit job ahead.)


      Queues with extended wall time limits

      are not available in general. This Queue spec1 is available for Jobs, which can not run within the 24h timeframe. Access to this queue is only granted by passing an evaluation process. Following rules apply to this queue:

      • Jobs may be killed for operation reasons at any time.
      • Jobs will be accounted in any case. This is also true if the job has to be terminated for operational reasons.
      • Joblimit per Group = 1
      • Joblimit per user = 1
      • Total number of nodes used for this queue = 64
      • Only general compute node types available, no visualisation or special node types
      • Low scheduling priority
      • Max walltime 96h