- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Batch System PBSPro (Hawk): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
No edit summary
Line 72: Line 72:
<br>
<br>
''Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.''
''Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.''


== Node types ==
== Node types ==
Line 78: Line 79:
* 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
* 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
* 1 x AMD EPYC Rome (2 x 64 cores each):  select with #PBS -l node_type=rome
* 1 x AMD EPYC Rome (2 x 64 cores each):  select with #PBS -l node_type=rome
== Core order ==
On Naples nodes, the core id corresponds as follows to hyperthreads and sockets:
core 0 - core 31: hyperthread 0 @ socket 0
core 32 - core 63: hyperthread 0 @ socket 1
core 64 - core 95: hyperthread 1 @ socket 0
core 96 - core 127: hyperthread 1 @ socket 1

Revision as of 12:04, 4 October 2019

The batch system on Hawk TDS is PBSPro 19.2.1. For general usage see the PBS User Guide (19.2.3)

At the moment the setup is basic and it works for the TDS only. More features, testing and productive like setup will be done in July.

Examples

Single node

Here is an example for a job using a single node:

qsub -l nodes=2,walltime=1:00:00


The basic commands to build and run a MPI program are:

module load mpt

mpicc hi.hpe.c -o hi.hpe

mpirun ./hi.hpe


simple batch script

Here is a simple pbs job script:

File: Job.hi.hpe.pbs
#!/bin/bash

#PBS -N Hi_Thomas
#PBS -l select=16:mpiprocs=64
#PBS -l walltime=00:20:00
 
module load mpt
mpirun ./hi.hpe


To submit the job script execute

qsub Job.hi.hpe.pbs


simple batch script for Hybrid MPI/OpenMP application

To run an MPI application with 64 Processes and two OpenMP threads per process on two compute nodes, include the following in the pbs job script:

File: Job.hi.mpiomp.hpe.pbs
#!/bin/bash

#PBS -N Hi_MPI_OpenMP
#PBS -l select=2:ncpus=64:mpiprocs=32:ompthreads=2
#PBS -l walltime=00:20:00
 
module load mpt
export OMP_NUM_THREADS=2
mpirun -np 64 omplace  -nt 2 [-vv] ./hi.mpiomp


The omplace command helps with the placement of OpenMP threads within an MPI program. In the above example, the threads in a 64-process MPI program with two threads per process are placed as follows:

  • Rank 0, thread 0 on CPU 0 of compute node 1
  • Rank 0, thread 1 on CPU 1 of compute node 1
  • ...
  • Rank 31, thread 1 on CPU 33 of compute node 1
  • Rank 32, thread 0 on CPU 0 of compute node 2
  • ...
  • Rank 63, thread 0 on CPU 32 of compute node 2
  • Rank 63, thread 1 on CPU 33 of compute node 2

The optional -vv parameter print out the placement of the processes and threads to standard output.
Warning: Due to the limited scaling of the standard output, you should not use the optional parameter -vv for medium and large jobs.


Node types

There are two types of nodes installed:

  • 16 x AMD EPYC Naples (2 x 32 cores each): select with #PBS -l node_type=naples
  • 1 x AMD EPYC Rome (2 x 64 cores each): select with #PBS -l node_type=rome


Core order

On Naples nodes, the core id corresponds as follows to hyperthreads and sockets: core 0 - core 31: hyperthread 0 @ socket 0 core 32 - core 63: hyperthread 0 @ socket 1 core 64 - core 95: hyperthread 1 @ socket 0 core 96 - core 127: hyperthread 1 @ socket 1