- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -
Batch System PBSPro (vulcan): Difference between revisions
(118 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
The only way to | Applications cannot be run directly from the login node on the compute nodes of the system. | ||
The only way to get access to the compute nodes of the system is to use the batch system. | |||
Therefore a batch script has to be provided, which includes all the necessary commands to run the application. | |||
The installed batch system is [https://www.pbspro.org/ PBSPro] | |||
You generally interact with the batch system in two ways: (1) through options specified in job submission scripts (these are detailed below in the examples) and (2) by using PBSPro commands on the login nodes. | |||
You generally interact with the batch system in two ways: through options specified in job submission scripts (these are detailed below in the examples) and by using PBSPro commands on the login nodes. | |||
There are three key commands used to interact with PBSPro: | |||
; qsub : used to submit your batch script to the system to be executed as a batch job | |||
; qstat : to display information about submitted batch jobs | |||
; qdel : to discard a submitted or running batch job | |||
Check the man page | Check the man pages of the individual commands as well as the man page for PBSPro on the login node for more information: | ||
{{Command|command=man qsub | |||
man qstat | |||
man qdel | |||
man pbs_professional | |||
}} | |||
If you are looking for in depth information consult the User Guide in the [https://www.altair.com/pbs-works-documentation/ PBS Pro documentation]. | |||
---- | ---- | ||
Line 45: | Line 57: | ||
<ol> | <ol> | ||
<font color=red><li><pre>select=<number of nodes>:<node_resource_variable=type></pre></li></font> | <font color=red><li><pre>select=<number of nodes>:<node_resource_variable=type></pre></li></font> | ||
<ul>To distinguish between different nodes 4 node resource variables are assigned to each node. The '''''node_type''''', '''''node_type_cpu''''', '''''node_type_mem''''' and '''''node_type_core''''' of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes. For some special nodes there are also the resources '''''localscratch''''' and '''''linkspeed''''' available. | <ul>To distinguish between different nodes 4 node resource variables are assigned to each node. The '''''node_type''''', '''''node_type_cpu''''', '''''node_type_mem''''' and '''''node_type_core''''' of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes. For some special nodes there are also the resources '''''localscratch''''' (see [[NEC_Cluster_Disk_Storage_(vulcan)#localscratch | localscratch ]]) and '''''linkspeed''''' available. | ||
{|border="1" cellpadding="2" | {|border="1" cellpadding="2" | ||
|+'''Available node types:''' | |+'''Available node types:''' | ||
|- | |- | ||
|'''node_type'''||'''node_type_cpu'''||'''node_type_mem'''||'''node_type_core'''|| '''Graphic'''||'''localscratch'''||'''linkspeed'''|||''' | |'''node_type'''||'''node_type_cpu'''||'''node_type_mem'''||'''node_type_core'''|| '''Graphic / Accelerator'''||'''localscratch'''||'''linkspeed'''|||'''describtion'''||'''notes'''||'''# of nodes (vulcan)'''||'''# of nodes (vulcan2) | ||
|- | |- | ||
|'' | |''clx-21''|| CascadeLake@2.10GHz || 384gb || 40c || || 1.8TB || HDR100 || Intel Xeon Gold 6230 @ 2.5GHz, CascadeLake, 384GB memory, 1.8TB /localscratch || 2 x 20 core-CPU per node<br>Entgeldordnung: CS500-Spark || 8 || 0 | ||
|- | |- | ||
|'' | |''clx-25, clx''|| CascadeLake@2.50GHz || 384gb || 40c || || || HDR100 || Intel Xeon Gold 6248 @ 2.5GHz, CascadeLake, 384GB memory || 2 x 20 core-CPU per node<br>Entgeldordnung: Cascadelake 384 GB || 96 || 0 | ||
|- | |- | ||
|'' | |''skl''|| Skylake@2.0GHz || 192gb || 40c || || || EDR || Intel Xeon Gold 6138 @ 2.00GHz, Skylake, 92GB memory || 2 x 20 core-CPU per node<br>Entgeldordnung: SKL-192GB || 72 || 0 | ||
|- | |- | ||
|'' | |''smp1500''|| Skylake@2.40GHz || 1536gb || 40c || || || HDR100 || Intel Gold 6148 @ 2.40GHz, Skylake, 1.5TByte memory || 2 x 20 core-CPU per node<br>will be shared with other jobs!<font color=red> Please use <tt>"qsub -q smp -l select=1:node_type=smp ..." </tt></font><br>Entgeldordnung: smp Knoten || 1 (shared)|| 0 | ||
|- | |- | ||
|''[[ | |''[[NEC_Aurora_HW| aurora]]''|| Skylake@2.60GHz || 192gb || 24c || 8x NEC Aurora vector CPUs|| || 2xEDR || Intel Xeon Gold 6126 | 2.60GHz, Skylake, 192GB memory, 8 NEC Aurora TSUBASA nodes || 2 x 12 core-CPU per node<br><font color=red> Please use <tt>"qsub -q vector -l select=1:node_type=aurora ..." </tt></font> || 8 || 0 | ||
|- | |- | ||
|'' | |''p100''|| Broadwell@3.00GHz || 1024gb || 24c || Nvidia Tesla P100 || 11TB || HDR100 || Cuda Node with Nvidia Tesla P100<br>Intel Xeon E5-2687Wv4 @ 3.0GHz, Broadwell, 1TB memory, 11TB local SSD scratch disk || 2 x 8 core-CPU per node<br>reserved for special users! <font color=red> Please use <tt>"qsub -q R_i1 -l select=1:node_type=p100 ..." </tt></font> || 3 || 0 | ||
|- | |- | ||
|'' | |''visamd''|| Skylake@2.60GHz || 96gb || 8c || AMD Radeon PRO WX 8200 || || HDR100 || Graphic Node with AMD Radeon PRO WX 8200<br> Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory || 2 x 4 core-CPU per node<br>only 1 node per job! <font color=red> Please use <tt>"qsub -q vis -l select=1:node_type=visamd ..." </tt></font><br>Entgeldordnung: Grafikserver AMD || 6 || 0 | ||
|- | |- | ||
|'' | |''visnv''|| Skylake@2.60GHz || 96gb || 8c || Nvidia Quadro RTX 4000 || || HDR100 || Graphic Node with Nvidia Quadro RTX 4000<br>Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory || 2 x 6 core-CPU per node<br>only 1 node per job! (1 node is reserved for special users) <font color=red> Please use <tt>"qsub -q vis -l select=1:node_type=visnv ..." </tt></font><br>Entgeldordnung: Grafikserver NVIDIA || 2 || 1 | ||
|- | |- | ||
|'' | |''genoa''|| Zen4@2.7GHz || 768gb || 64c || || || HDR100 || AMD compute node with 2x AMD Epyc 9334 Genoa, 2.7GHz base, 64 cores total, 768GB DDR5 || 2 x 32 core-CPU || 60 || 96 (1 is in shared mode) | ||
|- | |- | ||
|'' | |''genoa-smp''|| Zen4@2.7GHz || 3072gb || 64c || || || HDR100 || AMD shared node with 2x AMD Epyc 9334 Genoa, 2.7GHz base, 64 cores total, 3072GB DDR5 || 2 x 32 core-CPU per node<br>will be shared with other jobs!<font color=red> Please use <tt>"qsub -q smp -l select=1:node_type=smp ..." </tt></font> || 2 (shared) || 0 | ||
|- | |- | ||
|'' | |''genoa-a30''|| Zen4@3.0GHz || 768gb || 32c || Nvidia A30 with 24GB HBM2e || 3.8TB NVMe || HDR100 || GPU Node 2x AMD Epyc 9124 Genoa, 3.0GHz base, 32 cores total, 768GB DDR5, 3.8TB NVMe, 1x Nvidia A30 with 24GB HBM2e || 2 x 16 core-CPU || 24 || 0 | ||
|- | |- | ||
|'' | |''rome256gb32c'' || Zen2@3.0GHz || 256GB || 32c || || 1.8TB NVMe || HDR100 || || 2x 16 core-CPU || 3 || 0 | ||
|- | |- | ||
|'' | |''rome512gb48c-mi50'' || Zen2@2.3GHz || 512GB || 48c || 8x AMD Instinct Mi50 || 1.8TB NVMe || HDR100 || || 1x 48 core-CPU || 10 || 0 | ||
|- | |- | ||
|} | |} | ||
Line 87: | Line 94: | ||
'''Multi node type job can also be specified using a ''+'':''' | '''Multi node type job can also be specified using a ''+'':''' | ||
<ul> | <ul> | ||
<pre>select=1:node_type= | <pre>select=1:node_type=clx-25:node_type_mem=384gb+3:node_type=skl:node_type_mem=192gb:node_type_core=40c</pre> | ||
The example above will allocate 1 | The example above will allocate 1 clx-25 node with 384 GB memory and 3 skl nodes (the 40 cores type) with 192 GB memory. | ||
</ul> | </ul> | ||
Line 94: | Line 101: | ||
To allocate special nodes with local disk you can use the special node resource variable '''''localscratch''''': | To allocate special nodes with local disk you can use the special node resource variable '''''localscratch''''': | ||
<ul> | <ul> | ||
<pre>select=1:node_type= | <pre>select=1:node_type=clx-21:localscratch=1800GB</pre> | ||
</ul> | </ul> | ||
</ul> | </ul> | ||
Line 111: | Line 115: | ||
#!/bin/bash | #!/bin/bash | ||
#PBS -N job_name | #PBS -N job_name | ||
#PBS -l select=2:node_type= | #PBS -l select=2:node_type=clx:mpiprocs=40 | ||
#PBS -l walltime=00:20:00 | #PBS -l walltime=00:20:00 | ||
Line 126: | Line 130: | ||
qsub my_batchjob_script.pbs | qsub my_batchjob_script.pbs | ||
* Setting qsub options on the command line will overwrite the settings given in the batch script: | * Setting qsub options on the command line will overwrite the settings given in the batch script: | ||
qsub -N other_name -l select=2:node_type= | qsub -N other_name -l select=2:node_type=clx:mpiprocs=40 -l walltime=00:20:00 my_batchjob_script.pbs | ||
* The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available. | * The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available. | ||
* At the end of the execution output and error files are returned to your HOME directory | * At the end of the execution output and error files are returned to your HOME directory | ||
* This example will run your executable "my_mpi_executable" in parallel with | * This example will run your executable "my_mpi_executable" in parallel with 80 MPI processes (mpiprocs=40 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 40 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The '''mpirun''' example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 80 MPI processes with 40 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun. | ||
{{Note|text = | {{Note|text = | ||
<li> While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the | <li> While your job is running (in Batch Mode), STDOUT and STDERR are written to a file or files in a system directory and the output is copied to your submission directory (PBS_O_WORKDIR) only after the job completes. Specifying the | ||
<pre>qsub - | <pre>qsub -koed my_batchjob_script.pbs</pre> | ||
option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while | option here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.</li> | ||
}} | }} | ||
=== Interactive batch Mode === | === Interactive batch Mode === | ||
Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command: | Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command: | ||
qsub -I -l select=2:node_type= | qsub -I -l select=2:node_type=clx:ncpus=40:mpiprocs=40 -l walltime=00:30:00 | ||
If the requested resources are available and free (in the example above: 2 | If the requested resources are available and free (in the example above: 2 clx nodes/40 cores each, 30 minutes, prepared for 40 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources. | ||
Now you have to use the '''mpirun''' command to launch your parallel application to the allocated compute nodes. | Now you have to use the '''mpirun''' command to launch your parallel application to the allocated compute nodes. | ||
When you are finished, enter '''logout''' to exit the batch system and return to the normal command line. | When you are finished, enter '''logout''' to exit the batch system and return to the normal command line. | ||
=== PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus) === | === PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus) === | ||
Line 154: | Line 157: | ||
'''Example:''' | '''Example:''' | ||
qsub -l select=2:node_type= | qsub -l select=2:node_type=skl ./myscript | ||
The batch system allocates two node of type | The batch system allocates two node of type skl. The file <tt>${PBS_NODEFILE}</tt> contains: | ||
node1 | node1 | ||
node2 | node2 | ||
Line 163: | Line 166: | ||
'''select example with 2 chunk requests (seperated by '+'):''' | '''select example with 2 chunk requests (seperated by '+'):''' | ||
qsub -l select=2: | qsub -l select=2:skl:mpiprocs=2+1:node_type=skl:mpiprocs=3 ./myscript | ||
The batch system allocates 2 nodes of type | The batch system allocates 2 nodes of type skl each for 2 PE's and 1 node of type skl for 3 PE's. Then the file <tt>${PBS_NODEFILE}</tt> contains: | ||
node1 | node1 | ||
node1 | node1 | ||
Line 193: | Line 196: | ||
groupname of this project has to be passed in the group_list: | groupname of this project has to be passed in the group_list: | ||
<pre>qsub -l select=1:node_type= | <pre>qsub -l select=1:node_type=clx -W group_list=<groupname></pre> | ||
To get your available groups: | To get your available groups: | ||
Line 206: | Line 209: | ||
: E.g. a reservation of some nodes is bound to the queue named ''workday'': | : E.g. a reservation of some nodes is bound to the queue named ''workday'': | ||
qsub -q workday -l select=1:node_type= | qsub -q workday -l select=1:node_type=clx -l walltime=1:00 testjob.cmd | ||
Following reservations are availalble at the moment: | Following reservations are availalble at the moment: | ||
Line 214: | Line 217: | ||
|'''reservation'''||'''resources'''||'''queue'''||'''limitations'''||'''notes''' | |'''reservation'''||'''resources'''||'''queue'''||'''limitations'''||'''notes''' | ||
|- | |- | ||
| | |S150017||1 node of node_type clx (40 cores, 384GB)||workday for routing to S150017||valid on Mo-Fr 6:00-18:00, user- and group-run limits, walltime limit, nodecount limits will be set for jobs using this reservation||This daytime reservation should be used for jobs with a short walltime | ||
|} | |} | ||
Line 241: | Line 244: | ||
#!/bin/bash | #!/bin/bash | ||
# | # | ||
# Simple PBS batch script that reserves two exclusive | # Simple PBS batch script that reserves two exclusive Skylake nodes | ||
# and runs only one MPI process on each node (in total 2 MPI processes) | # and runs only one MPI process on each node (in total 2 MPI processes) | ||
# The walltime is 10min | # The walltime is 10min | ||
# | # | ||
#PBS -l select=2:node_type= | #PBS -l select=2:node_type=skl:mpiprocs=1 | ||
#PBS -l walltime=00:10:00 | #PBS -l walltime=00:10:00 | ||
Line 272: | Line 275: | ||
#!/bin/bash | #!/bin/bash | ||
# | # | ||
# Simple PBS batch script that reserves two exclusive | # Simple PBS batch script that reserves two exclusive Skylake nodes | ||
# and runs four MPI process on each node (in total 8 MPI processes) | # and runs four MPI process on each node (in total 8 MPI processes) | ||
# The walltime is 10min | # The walltime is 10min | ||
# | # | ||
#PBS -l select=2:node_type= | #PBS -l select=2:node_type=skl:mpiprocs=4 | ||
#PBS -l walltime=00:10:00 | #PBS -l walltime=00:10:00 | ||
Line 297: | Line 300: | ||
# | # | ||
# Simple PBS batch script that runs a scalar job | # Simple PBS batch script that runs a scalar job | ||
# on 1 | # on 1 Skylake node using 2h | ||
# | # | ||
#PBS -l select=1:node_type= | #PBS -l select=1:node_type=skl,walltime=2:00:00 | ||
cd $PBS_O_WORKDIR | cd $PBS_O_WORKDIR | ||
./my_executable | ./my_executable | ||
Line 315: | Line 318: | ||
<BR> | <BR> | ||
<LI> | <LI> | ||
Starting a script using 3 nodes of node_type ' | Starting a script using 3 nodes of node_type 'skl' and a real time of 2 hours: | ||
<pre>qsub -l select=3:node_type= | <pre>qsub -l select=3:node_type=skl,walltime=2:00:00 <script></pre> | ||
</li> | </li> | ||
<BR> | <BR> | ||
<li> | <li> | ||
Starting a script using | Starting a script using 5 cluster nodes of node_type 'clx' using 10 processors on each node: | ||
<pre>qsub -l select= | <pre>qsub -l select=5:node_type=clx:mpiprocs=10,walltime=2:00:00 <script></pre> | ||
</li> | </li> | ||
<BR> | <BR> | ||
<li> | <li> | ||
Starting a interactive batch job using 5 | Starting a interactive batch job using 5 clx nodes with a job real time of 300 seconds: | ||
<pre>qsub -I -l select=5:node_type= | <pre>qsub -I -l select=5:node_type=clx,walltime=300</pre> | ||
{{Note|text = | {{Note|text = | ||
For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command | For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the command | ||
Line 345: | Line 338: | ||
<pre>qstat -n <jobid></pre> on the master node. | <pre>qstat -n <jobid></pre> on the master node. | ||
You can log in from the frontend or any assigned node to all other assigned nodes by | You can log in from the frontend or any assigned node to all other assigned nodes by | ||
<pre>ssh <nodename></pre> | <pre>export PBS_JOBID=<jobid> | ||
ssh <nodename></pre> | |||
<jobid> is of format (number.batchserver e.g: 123456.cl1intern). | |||
If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated. | If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated. | ||
}} | }} | ||
Line 352: | Line 347: | ||
<BR> | <BR> | ||
<li> | <li> | ||
Other possibilities to request cluster nodes. You want 4 nodes with a minimum of | Other possibilities to request cluster nodes. You want 4 nodes with a minimum of 40 cores regardless which type of nodes. And you want to run 40 MPI processes on each of the 4 nodes: | ||
<pre>qsub -l select=4:ncpus= | <pre>qsub -l select=4:ncpus=40:mpiprocs=40 <script></pre> | ||
Then its possible you will get a mix of 4 nodes composed of | Then its possible you will get a mix of 4 nodes composed of clx and skl node types. | ||
</li> | </li> | ||
Line 367: | Line 358: | ||
<pre>id</pre> | <pre>id</pre> | ||
Choose a valid groupname for your job (abc12345 will serve as a placeholder here): | Choose a valid groupname for your job (abc12345 will serve as a placeholder here): | ||
<pre>qsub -l select=5:node_type= | <pre>qsub -l select=5:node_type=clx,walltime=300 -W group_list=abc12345</pre> | ||
</li> | </li> | ||
</UL> | </UL> | ||
Line 378: | Line 369: | ||
qstat [options] | qstat [options] | ||
nstat | nstat | ||
batchstat [options] | |||
</pre> | </pre> | ||
For detailed informations, see man pages: | For detailed informations, see man pages: | ||
<pre> | <pre> | ||
nstat -h | nstat -h | ||
batchstat -h | |||
man qstat | man qstat | ||
man pbsnodes | man pbsnodes | ||
Line 390: | Line 383: | ||
'''Examples''' | '''Examples''' | ||
<ul> | <ul> | ||
list all batch jobs: | list all your own batch jobs: | ||
<font color=red>(query status of jobs owned by other users has been prohibited!)</font> | |||
<pre>qstat -a</pre> | <pre>qstat -a</pre> | ||
list all batch jobs (anonymous) | |||
<pre>batchstat</pre> | |||
lists all batch queues with resource limit settings: | lists all batch queues with resource limit settings: | ||
<pre>qstat -q</pre> | <pre>qstat -q</pre> | ||
lists node information of | lists node information of one of your batch job ID: | ||
<pre>qstat -n <JOB_ID></pre> | <pre>qstat -n <JOB_ID></pre> | ||
lists detailed information of | lists detailed information of one of your batch job ID: | ||
<pre>qstat -f <JOB_ID></pre> | <pre>qstat -f <JOB_ID></pre> | ||
Displays estimated start time for queued jobs | Displays estimated start time for your queued jobs | ||
<pre>qstat -T <JOB_ID></pre> | <pre>qstat -T <JOB_ID></pre> | ||
Displays status information for jobs, job arrays, and subjobs: | Displays status information for your jobs, job arrays, and subjobs: | ||
<pre>qstat -t <JOB_ID></pre> | <pre>qstat -t <JOB_ID></pre> | ||
lists information of the PBS node status: | lists information of the PBS node status: | ||
Line 410: | Line 406: | ||
<pre> | <pre> | ||
nstat | nstat | ||
batchstat | |||
</pre> | </pre> | ||
== ssh from login nodes to your allocated nodes of your job == | |||
You can only connect to nodes that belongs to you, that are, the allocated nodes of your jobs. | |||
To log in to this nodes via ssh from the login nodes (frontend nodes) you have to set the environment PBS_JOBID. | |||
First find your running jobid's and the nodes which belongs to the jobs: | |||
qstat -rnw | |||
Next step is setting the environment PBS_JOBID on the login node: | |||
export PBS_JOBID=<JOB ID> | |||
(the <JOB ID> is in form of 123456.cl5intern)<BR> | |||
Now you are able to login form login node via ssh to the allocated nodes of your job with the corresponding jobid. | |||
== DISPLAY: X11 applications on interactive batch jobs == | == DISPLAY: X11 applications on interactive batch jobs == | ||
Line 427: | Line 434: | ||
}} | }} | ||
=== X server on compute nodes === | |||
In general, no X server process is running on the compute nodes! | |||
For example to run a window manger on the compute nodes like icewm you need an X server. | |||
Some visualisation nodes have a graphic card installed (see [[Batch_System_PBSPro_(vulcan)#Node_types | here]]) on which an X server can be started inside the batch job by your self: | |||
HLRS-startX | |||
This tool start an X server and returns the DISPLAY port on which the server is started. | |||
You can also use: | |||
get-X-display-port | |||
to get the DISPLAY port for your running X server. | |||
To use the X server in your application you have to use the environment: | |||
export DISPLAY=:<port> | |||
Alternatively see [[Graphic_Environment]] | |||
== Container, singularity, AI == | |||
Solving problems using artificial intelligence, usually you need nodes equipped with fast local disks and GPU's. | |||
Some nodes (node_type=clx-ai) are able to run singularity containers. | |||
To be able to start singularity container, the nodes have to be prepared before your batch job starts. | |||
You can do this by a resource request ('UNS=True') at your batch submission: | |||
qsub -l select=1:node_type=clx-ai:UNS=True,walltime=300 <mybatchjob> | |||
Note: Only the nodes with node_type=clx-ai are able to run singularity container! | |||
More about singularity container software can be found here: [[Urika CS|Cray Urika CS software stack ]] | |||
== NEC vector CPU's Aurora == | |||
To use the NEC vector CPU's you need to submit your jobs to a special Queue "vector": | |||
qsub -q vector -l select=1:node_type=aurora,walltime=300 <mybatchjob> | |||
More Details about NEC Auora can be found [[Aurora_Tsubasa| here (Aurora Tsubasa Platform)]]. | |||
== Queue Policies and Limitations == | == Queue Policies and Limitations == | ||
Line 438: | Line 475: | ||
For larger jobs or for special job requirements different restrictions are in place respectively you have to consult the project team | For larger jobs or for special job requirements different restrictions are in place respectively you have to consult the project team | ||
Each limit settings and policies could be changed in future to adjust the cluster usage for new user requirements. | |||
At the moment following queues and policies are defined: | |||
=== route (default) === | === route (default) === | ||
If users don't declare a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs (academic users) are: | If users don't declare a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs (academic users) are: | ||
Line 464: | Line 505: | ||
|walltime|| || 24 hours || | |walltime|| || 24 hours || | ||
|- | |- | ||
|available nodes|| 2 || | |available nodes|| 2 || 200 per job (250 in total) || | ||
|- | |- | ||
|priority|| || || normal | |priority|| || || normal | ||
Line 479: | Line 520: | ||
|'''resource'''||'''min'''||'''max''' || '''note''' | |'''resource'''||'''min'''||'''max''' || '''note''' | ||
|- | |- | ||
|walltime|| || | |walltime|| || 25 minutes || | ||
|- | |- | ||
|available nodes|| || | |available nodes|| || 40 || | ||
|- | |- | ||
|priority|| || || very high | |priority|| || || very high | ||
Line 527: | Line 568: | ||
* User limits: | * User limits: | ||
** limited number of jobs of one user that can run at the same time | ** limited number of jobs of one user that can run at the same time | ||
** in total a user can only allocate | ** in total a user can only allocate 250 nodes. | ||
* User Group limits: | * User Group limits: | ||
** limited number of jobs of users in the same group that can run at the same time | ** limited number of jobs of users in the same group that can run at the same time | ||
Line 534: | Line 575: | ||
** The number of jobs for each user in the different job queues are restricted. If you reach this number you can submit further jobs when prior jobs have ended. | ** The number of jobs for each user in the different job queues are restricted. If you reach this number you can submit further jobs when prior jobs have ended. | ||
*** (If more jobs are submitted than allowed for one job queue the old ones will be placed in the dispatcher queue 'route' and will move up in the proper destination queue after jobs from this user in the corresponding queue have ended. The waiting queue for each user will take up to 10 jobs. With this it is possible to submit job ahead.) | *** (If more jobs are submitted than allowed for one job queue the old ones will be placed in the dispatcher queue 'route' and will move up in the proper destination queue after jobs from this user in the corresponding queue have ended. The waiting queue for each user will take up to 10 jobs. With this it is possible to submit job ahead.) | ||
=== Queues with extended wall time limits === | |||
are not available in general. This Queue '''spec1''' is available for Jobs, which can not run within the 24h timeframe. Access to this queue is only granted by passing an evaluation process. Following rules apply to this queue: | |||
* Jobs may be killed for operation reasons at any time. | |||
* Jobs will be accounted in any case. This is also true if the job has to be terminated for operational reasons. | |||
* Joblimit per Group = 1 | |||
* Joblimit per user = 1 | |||
* Total number of nodes used for this queue = 64 | |||
* Only general compute node types available, no visualisation or special node types | |||
* Low scheduling priority | |||
* Max walltime 96h |
Latest revision as of 10:00, 26 August 2024
Introduction
Applications cannot be run directly from the login node on the compute nodes of the system. The only way to get access to the compute nodes of the system is to use the batch system. Therefore a batch script has to be provided, which includes all the necessary commands to run the application. The installed batch system is PBSPro
You generally interact with the batch system in two ways: (1) through options specified in job submission scripts (these are detailed below in the examples) and (2) by using PBSPro commands on the login nodes.
There are three key commands used to interact with PBSPro:
- qsub
- used to submit your batch script to the system to be executed as a batch job
- qstat
- to display information about submitted batch jobs
- qdel
- to discard a submitted or running batch job
Check the man pages of the individual commands as well as the man page for PBSPro on the login node for more information:
man qstat
man qdel
man pbs_professional
If you are looking for in depth information consult the User Guide in the PBS Pro documentation.
Requesting Resources using batch system
Resources are allocated to jobs both by explicitly requesting them and by applying specified defaults.
Jobs explicitly request resources either at the host level in chunks defined in a selection statement, or in job-wide resource requests.
-
Format:
- job wide request:
qsub ... -l <resource name>=<value>
The only resources that can be in a job-wide request are server-level or queue-level resources, such as walltime.
- selection statement:
qsub ... -l select=<chunks>
The only resources that can be requested in chunks are host-level resources, such as mem and ncpus. A chunk is the smallest set of resources that will be allocated to a job. It is one or more resource_name=value statements separated by a colon, e.g.:
ncpus=2:mem=32GB
A selection statement is of the form:
-l select=[N:]chunk[+[N:]chunk ...]
Note: If N is not specified, it is taken to be 1. No spaces are allowed between chunks.
Node types
You have to specify the resources you need for your batch job. These resources are specified by including them in the -l argument (selection statement and job-wide resources) on the qsub command or in the PBS job script. The 2 important resources you have to specify are number of nodes of a specific node type in the selection statement and the walltime in the job-wide resource request you need for this job:
select=<number of nodes>:<node_resource_variable=type>
walltime=<time>
- To distinguish between different nodes 4 node resource variables are assigned to each node. The node_type, node_type_cpu, node_type_mem and node_type_core of each node. You have to specify at least one of the resource variable or you can specify a valid available combination of the resources for a specific type of nodes. For some special nodes there are also the resources localscratch (see localscratch ) and linkspeed available.
node_type | node_type_cpu | node_type_mem | node_type_core | Graphic / Accelerator | localscratch | linkspeed | describtion | notes | # of nodes (vulcan) | # of nodes (vulcan2) |
clx-21 | CascadeLake@2.10GHz | 384gb | 40c | 1.8TB | HDR100 | Intel Xeon Gold 6230 @ 2.5GHz, CascadeLake, 384GB memory, 1.8TB /localscratch | 2 x 20 core-CPU per node Entgeldordnung: CS500-Spark |
8 | 0 | |
clx-25, clx | CascadeLake@2.50GHz | 384gb | 40c | HDR100 | Intel Xeon Gold 6248 @ 2.5GHz, CascadeLake, 384GB memory | 2 x 20 core-CPU per node Entgeldordnung: Cascadelake 384 GB |
96 | 0 | ||
skl | Skylake@2.0GHz | 192gb | 40c | EDR | Intel Xeon Gold 6138 @ 2.00GHz, Skylake, 92GB memory | 2 x 20 core-CPU per node Entgeldordnung: SKL-192GB |
72 | 0 | ||
smp1500 | Skylake@2.40GHz | 1536gb | 40c | HDR100 | Intel Gold 6148 @ 2.40GHz, Skylake, 1.5TByte memory | 2 x 20 core-CPU per node will be shared with other jobs! Please use "qsub -q smp -l select=1:node_type=smp ..." Entgeldordnung: smp Knoten |
1 (shared) | 0 | ||
aurora | Skylake@2.60GHz | 192gb | 24c | 8x NEC Aurora vector CPUs | 2xEDR | 2.60GHz, Skylake, 192GB memory, 8 NEC Aurora TSUBASA nodes | 2 x 12 core-CPU per node Please use "qsub -q vector -l select=1:node_type=aurora ..." |
8 | 0 | |
p100 | Broadwell@3.00GHz | 1024gb | 24c | Nvidia Tesla P100 | 11TB | HDR100 | Cuda Node with Nvidia Tesla P100 Intel Xeon E5-2687Wv4 @ 3.0GHz, Broadwell, 1TB memory, 11TB local SSD scratch disk |
2 x 8 core-CPU per node reserved for special users! Please use "qsub -q R_i1 -l select=1:node_type=p100 ..." |
3 | 0 |
visamd | Skylake@2.60GHz | 96gb | 8c | AMD Radeon PRO WX 8200 | HDR100 | Graphic Node with AMD Radeon PRO WX 8200 Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory |
2 x 4 core-CPU per node only 1 node per job! Please use "qsub -q vis -l select=1:node_type=visamd ..." Entgeldordnung: Grafikserver AMD |
6 | 0 | |
visnv | Skylake@2.60GHz | 96gb | 8c | Nvidia Quadro RTX 4000 | HDR100 | Graphic Node with Nvidia Quadro RTX 4000 Intel Xeon Silver 4112 @ 2.6GHz, Skylake, 96GB memory |
2 x 6 core-CPU per node only 1 node per job! (1 node is reserved for special users) Please use "qsub -q vis -l select=1:node_type=visnv ..." Entgeldordnung: Grafikserver NVIDIA |
2 | 1 | |
genoa | Zen4@2.7GHz | 768gb | 64c | HDR100 | AMD compute node with 2x AMD Epyc 9334 Genoa, 2.7GHz base, 64 cores total, 768GB DDR5 | 2 x 32 core-CPU | 60 | 96 (1 is in shared mode) | ||
genoa-smp | Zen4@2.7GHz | 3072gb | 64c | HDR100 | AMD shared node with 2x AMD Epyc 9334 Genoa, 2.7GHz base, 64 cores total, 3072GB DDR5 | 2 x 32 core-CPU per node will be shared with other jobs! Please use "qsub -q smp -l select=1:node_type=smp ..." |
2 (shared) | 0 | ||
genoa-a30 | Zen4@3.0GHz | 768gb | 32c | Nvidia A30 with 24GB HBM2e | 3.8TB NVMe | HDR100 | GPU Node 2x AMD Epyc 9124 Genoa, 3.0GHz base, 32 cores total, 768GB DDR5, 3.8TB NVMe, 1x Nvidia A30 with 24GB HBM2e | 2 x 16 core-CPU | 24 | 0 |
rome256gb32c | Zen2@3.0GHz | 256GB | 32c | 1.8TB NVMe | HDR100 | 2x 16 core-CPU | 3 | 0 | ||
rome512gb48c-mi50 | Zen2@2.3GHz | 512GB | 48c | 8x AMD Instinct Mi50 | 1.8TB NVMe | HDR100 | 1x 48 core-CPU | 10 | 0 |
Multi node type job can also be specified using a +:
select=1:node_type=clx-25:node_type_mem=384gb+3:node_type=skl:node_type_mem=192gb:node_type_core=40c
The example above will allocate 1 clx-25 node with 384 GB memory and 3 skl nodes (the 40 cores type) with 192 GB memory.
To allocate special nodes with local disk you can use the special node resource variable localscratch:
select=1:node_type=clx-21:localscratch=1800GB
Batch Mode
Production jobs are typically run in batch mode. Batch scripts are shell scripts containing flags and commands to be interpreted by a shell and are used to run a set of commands in sequence.
- The number of required nodes, cores, wall time and more can be determined by the parameters in the job script header with "#PBS" before any executable commands in the script.
#!/bin/bash #PBS -N job_name #PBS -l select=2:node_type=clx:mpiprocs=40 #PBS -l walltime=00:20:00 # Change to the direcotry that the job was submitted from cd $PBS_O_WORKDIR # using the INTEL MPI module module load mpi/impi # Launch the parallel mpi application (compiled with intel mpi) to the allocated compute nodes mpirun -np 48 ./my_mpi_executable arg1 arg2 > my_output_file 2>&1
- The job is submitted by the qsub command (all script head parameters #PBS can also be adjusted directly by qsub command options).
qsub my_batchjob_script.pbs
- Setting qsub options on the command line will overwrite the settings given in the batch script:
qsub -N other_name -l select=2:node_type=clx:mpiprocs=40 -l walltime=00:20:00 my_batchjob_script.pbs
- The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.
- At the end of the execution output and error files are returned to your HOME directory
- This example will run your executable "my_mpi_executable" in parallel with 80 MPI processes (mpiprocs=40 is the number of MPI processes on each node) . The batch system will allocate 2 nodes to your job for a maximum time of 20 minutes and place 40 processes on each node. The batch systems allocates nodes exclusively only for one job. After the walltime limit is exceeded, the batch system will terminate your job. The mpirun example above will start the parallel executable "my_mpi_executable" with the arguments "arg1" and "arg2". The job will be started using 80 MPI processes with 40 processes placed on each of your allocated nodes. You need to have nodes allocated by the batch system (qsub) before starting mpirun.
qsub -koed my_batchjob_script.pbsoption here and redirecting the output to a file (see example above) makes it possible for you to view STDOUT and STDERR of your job scripts while the job is running.
Interactive batch Mode
Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the "qsub -I" command:
qsub -I -l select=2:node_type=clx:ncpus=40:mpiprocs=40 -l walltime=00:30:00
If the requested resources are available and free (in the example above: 2 clx nodes/40 cores each, 30 minutes, prepared for 40 mpi processes on each node), then you will get a new session on the jobs head node for your requested resources. Now you have to use the mpirun command to launch your parallel application to the allocated compute nodes. When you are finished, enter logout to exit the batch system and return to the normal command line.
PBS_NODEFILE (MPI usage of multi-socket nodes and multi-core cpus)
In most MPI environments, the PBS_NODEFILE will be usefull to start the correct number of mpi processes on each allocated node. The jobs ${PBS_NODEFILE} contents depends on the number of MPI processes for each requested chunk. Inside a select statement of each chunk you can define a mpiprocs option (Type: integer). The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.
Example:
qsub -l select=2:node_type=skl ./myscript
The batch system allocates two node of type skl. The file ${PBS_NODEFILE} contains:
node1 node2
If the chunk request has the option mpiprocs defined, then it is possible to allocate the defined PE's on a node. This option especially allow OpenMPI to place the MPI processes of ranks on a shared node or alternatively on distributed nodes.
select example with 2 chunk requests (seperated by '+'):
qsub -l select=2:skl:mpiprocs=2+1:node_type=skl:mpiprocs=3 ./myscript
The batch system allocates 2 nodes of type skl each for 2 PE's and 1 node of type skl for 3 PE's. Then the file ${PBS_NODEFILE} contains:
node1 node1 node2 node2 node3 node3 node3
Defaults for Ressource Requests
If you don't set the resources for your job request, then you will get default resource limits for your job.
resource | value | notes |
select | 1 | |
mpiprocs | 1 |
Please select your resource requests carefull.
To have the same environmental settings (exported environment) of your current session in your batchjob, the qsub command needs the option argument -V.
Run job on other Account ID
There are Unix groups associated to the project account ID (ACID). To run a job on a non-default project budget, the groupname of this project has to be passed in the group_list:
qsub -l select=1:node_type=clx -W group_list=<groupname>
To get your available groups:
id -Gn
Usage of a Reservation
For nodes which are reserved for special groups or users, you need to specify additional the queue which is intended for this reservation:
- E.g. a reservation of some nodes is bound to the queue named workday:
qsub -q workday -l select=1:node_type=clx -l walltime=1:00 testjob.cmd
Following reservations are availalble at the moment:
reservation | resources | queue | limitations | notes |
S150017 | 1 node of node_type clx (40 cores, 384GB) | workday for routing to S150017 | valid on Mo-Fr 6:00-18:00, user- and group-run limits, walltime limit, nodecount limits will be set for jobs using this reservation | This daytime reservation should be used for jobs with a short walltime |
Job Arrays
Job arrays are groups of similar jobs. Those jobs usually have slightly different parameters which depend on the current job index. This job index will be available in the $PBS_ARRAY_INDEX variable, which can be used in job scripts to calculate or generate any kind of job-specific (input)data.
Job arrays can be requested with
qsub -J <range> <my_array_jobscript>
range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1.
Examples
Examples for PBS options in job scripts
-
You can submit batch jobs using qsub. A very simple qsub script for a MPI job with PBSPro directives (#PBS ...) for the options of qsub looks like this:
#!/bin/bash # # Simple PBS batch script that reserves two exclusive Skylake nodes # and runs only one MPI process on each node (in total 2 MPI processes) # The walltime is 10min # #PBS -l select=2:node_type=skl:mpiprocs=1 #PBS -l walltime=00:10:00 ### go to directory where your job request was submitted cd $PBS_O_WORKDIR ### load for example the Intel MPI environment module load mpi/impi ### run you parallel application on the allocated nodes mpirun -np 2 ./mpitest
Warning:
- you have to specify a shell in the first line of your batch script
- you have to specify the number of nodes you need and the node type
- you have to specify the walltime the job needs
- allocated nodes will not be shared with other jobs, even though you uses the nodes only partial
-
If you want to use four MPI processes on each node this can be done like this:
#!/bin/bash # # Simple PBS batch script that reserves two exclusive Skylake nodes # and runs four MPI process on each node (in total 8 MPI processes) # The walltime is 10min # #PBS -l select=2:node_type=skl:mpiprocs=4 #PBS -l walltime=00:10:00 ### go to directory where your job request was submitted cd $PBS_O_WORKDIR ### load for example the Intel MPI environment module load mpi/impi ### run you parallel application on the allocated nodes mpirun -np 8 ./mpitest
-
If you need 2h wall time and one node you can use the following script:
#!/bin/bash # # Simple PBS batch script that runs a scalar job # on 1 Skylake node using 2h # #PBS -l select=1:node_type=skl,walltime=2:00:00 cd $PBS_O_WORKDIR ./my_executable
Examples for starting batch jobs:
-
Starting a script with all options specified inside the script file
qsub <script>
-
Starting a script using 3 nodes of node_type 'skl' and a real time of 2 hours:
qsub -l select=3:node_type=skl,walltime=2:00:00 <script>
-
Starting a script using 5 cluster nodes of node_type 'clx' using 10 processors on each node:
qsub -l select=5:node_type=clx:mpiprocs=10,walltime=2:00:00 <script>
-
Starting a interactive batch job using 5 clx nodes with a job real time of 300 seconds:
qsub -I -l select=5:node_type=clx,walltime=300
Note: For interactive Batch jobs, you don't need a script.sh file. If the requested resources are available, you will get an interactive shell on one of the allocated compute nodes. Which nodes are allocated can be shown with the commandcat $PBS_NODEFILE
on the batch job shell or with the PBS status command
qstat -n <jobid>
on the master node.You can log in from the frontend or any assigned node to all other assigned nodes by
export PBS_JOBID=<jobid> ssh <nodename>
<jobid> is of format (number.batchserver e.g: 123456.cl1intern).
If you exit the automatically established interactive shell to the node, it will be assumed that you finished your job and all other connections to the nodes will be terminated. -
Other possibilities to request cluster nodes. You want 4 nodes with a minimum of 40 cores regardless which type of nodes. And you want to run 40 MPI processes on each of the 4 nodes:
qsub -l select=4:ncpus=40:mpiprocs=40 <script>
Then its possible you will get a mix of 4 nodes composed of clx and skl node types.
-
Starting a script which should run on other Account ID:
First you have to know which Account ID's (groupnames) are valid for your login:
id
Choose a valid groupname for your job (abc12345 will serve as a placeholder here):
qsub -l select=5:node_type=clx,walltime=300 -W group_list=abc12345
Get the batch job status
qstat [options] nstat batchstat [options]
For detailed informations, see man pages:
nstat -h batchstat -h man qstat man pbsnodes
-
list all your own batch jobs:
(query status of jobs owned by other users has been prohibited!)
qstat -a
list all batch jobs (anonymous)
batchstat
lists all batch queues with resource limit settings:
qstat -q
lists node information of one of your batch job ID:
qstat -n <JOB_ID>
lists detailed information of one of your batch job ID:
qstat -f <JOB_ID>
Displays estimated start time for your queued jobs
qstat -T <JOB_ID>
Displays status information for your jobs, job arrays, and subjobs:
qstat -t <JOB_ID>
lists information of the PBS node status:
pbsnodes -a pbsnodes -l
gives informatioin of PBS node and job status:
nstat batchstat
ssh from login nodes to your allocated nodes of your job
You can only connect to nodes that belongs to you, that are, the allocated nodes of your jobs. To log in to this nodes via ssh from the login nodes (frontend nodes) you have to set the environment PBS_JOBID. First find your running jobid's and the nodes which belongs to the jobs:
qstat -rnw
Next step is setting the environment PBS_JOBID on the login node:
export PBS_JOBID=<JOB ID>
(the <JOB ID> is in form of 123456.cl5intern)
Now you are able to login form login node via ssh to the allocated nodes of your job with the corresponding jobid.
DISPLAY: X11 applications on interactive batch jobs
For X11 applications you need to have SSH X11 Forwarding enabled. This is usually activated per default. But to be sure you can set 'ForwardX11 yes' in your $HOME/.ssh/config. To have the same DISPLAY of your current session in your batchjob, the qsub command needs the option argument -X.
frontend> qsub -q vis -l select=1:node_type=fx5800,walltime=300 -X -I
X server on compute nodes
In general, no X server process is running on the compute nodes! For example to run a window manger on the compute nodes like icewm you need an X server. Some visualisation nodes have a graphic card installed (see here) on which an X server can be started inside the batch job by your self:
HLRS-startX
This tool start an X server and returns the DISPLAY port on which the server is started. You can also use:
get-X-display-port
to get the DISPLAY port for your running X server. To use the X server in your application you have to use the environment:
export DISPLAY=:<port>
Alternatively see Graphic_Environment
Container, singularity, AI
Solving problems using artificial intelligence, usually you need nodes equipped with fast local disks and GPU's. Some nodes (node_type=clx-ai) are able to run singularity containers. To be able to start singularity container, the nodes have to be prepared before your batch job starts. You can do this by a resource request ('UNS=True') at your batch submission:
qsub -l select=1:node_type=clx-ai:UNS=True,walltime=300 <mybatchjob>
Note: Only the nodes with node_type=clx-ai are able to run singularity container!
More about singularity container software can be found here: Cray Urika CS software stack
NEC vector CPU's Aurora
To use the NEC vector CPU's you need to submit your jobs to a special Queue "vector":
qsub -q vector -l select=1:node_type=aurora,walltime=300 <mybatchjob>
More Details about NEC Auora can be found here (Aurora Tsubasa Platform).
Queue Policies and Limitations
Different job queues are available for efficient resource usage.
In most cases users do not need to declare a job queue with the qsub command. Jobs are sorted to the right class automatically. In the following the definition for each job queue is given. In general jobs with a Duration up to 24 hours and half of the available resources can be submitted. Some special resources like nodes with graphics or very large memory, (job sharing nodes) are only available for special queues which have to be declared with the qsub command. For larger jobs or for special job requirements different restrictions are in place respectively you have to consult the project team
Each limit settings and policies could be changed in future to adjust the cluster usage for new user requirements.
At the moment following queues and policies are defined:
route (default)
If users don't declare a queue on qsub submission, then the jobs default queue will be this. The "route" queue is a routing queue with final destinations for the industrial user jobs and the standard jobs depends on users/groups and the requested resources. The destination queues of "route" for standard jobs (academic users) are:
single
This queue is available for all single node jobs.
resource | min | max | note |
walltime | 24 hours | ||
available nodes | 1 per job (64 in total) | only single node jobs | |
priority | low | ||
joblimit | 20 per user, 30 per group |
normal
This is for all regular parallel jobs using 2 nodes and more.
resource | min | max | note |
walltime | 24 hours | ||
available nodes | 2 | 200 per job (250 in total) | |
priority | normal | ||
joblimit | 20 per user |
test
This queue is for tests and development with restricted resources needs. The jobs in this queue are expected to deliver results after very short time. It's forbidden to use this queue for production jobs. Users have to declare this queue with the qsub submission.
resource | min | max | note |
walltime | 25 minutes | ||
available nodes | 40 | ||
priority | very high | ||
joblimit | 1 per user (4 for ALL) |
interactive
This queue is also only for batch jobs in interactive batch mode which can also be for tests and development. Users can not declare this queue with the qsub submission. But all interactive batch jobs will be routed to this queue automatically.
resource | min | max | note |
walltime | 8 hours | ||
available nodes | 32 | only for job in interactive batch mode | |
priority | very high | ||
joblimit | 2 per user |
vis
This queue is only available for jobs using graphic nodes. It can also be used in interactive batch mode. Only the nodes with a graphic card installed are available with this queue. Users have to declare this queue with the qsub job submission.
resource | min | max | note |
walltime | 10 hours | ||
available nodes | 1 per job (4 in total) | only nodes with graphic cards | |
priority | high | ||
joblimit | 2 per user, 4 per group |
Job Run Limitations
- The maximum time limit for a Job is 24hours.
- User limits:
- limited number of jobs of one user that can run at the same time
- in total a user can only allocate 250 nodes.
- User Group limits:
- limited number of jobs of users in the same group that can run at the same time
- Batch Queue limits of all user jobs:
- not all nodes / node types are available on each queue (visualisation nodes can not be used in multi node job queues)
- The number of jobs for each user in the different job queues are restricted. If you reach this number you can submit further jobs when prior jobs have ended.
- (If more jobs are submitted than allowed for one job queue the old ones will be placed in the dispatcher queue 'route' and will move up in the proper destination queue after jobs from this user in the corresponding queue have ended. The waiting queue for each user will take up to 10 jobs. With this it is possible to submit job ahead.)
Queues with extended wall time limits
are not available in general. This Queue spec1 is available for Jobs, which can not run within the 24h timeframe. Access to this queue is only granted by passing an evaluation process. Following rules apply to this queue:
- Jobs may be killed for operation reasons at any time.
- Jobs will be accounted in any case. This is also true if the job has to be terminated for operational reasons.
- Joblimit per Group = 1
- Joblimit per user = 1
- Total number of nodes used for this queue = 64
- Only general compute node types available, no visualisation or special node types
- Low scheduling priority
- Max walltime 96h