HPE Hawk

Hawk is the next generation HPC system at HLRS. It will replace the existing HazelHen system. The installation is planed to take place in Q4 2019. For more detailed information see the Hawk installation schedule.

This Page is under construction!

Batch System PBSPro (Hawk)

TODO

• 2019-08-07, dick@hlrs: hmpt always uses GCC (the 1st in PATH) -> adapt s.t. other compilers can also be used. Or do we have to use -lmpi instead?

• 2019-08-07, dick@hlrs: hmpt is ABI-compatible with MPICH-derivativs, but not so mpt
  • user should know about this!
  • @HPE: is hmpt a MPICH-derivative, but not so mpt?

• 2019-08-07, dick@hlrs: unclear that (h)mpt provides MPI lib -> call it "mpi/hmpt" and "mpi/mpt" instead?

• 2019-08-07, dick@hlrs: remove MPI delivered with RHEL

• 2019-08-07, dick@hlrs: Intel loads gcc module -> use (LD_)LIBRARY_PATH intead

• 2019-08-07, dick@hlrs: be careful: cc points to /usr/bin/gcc!

• 2019-08-14, khabi/offenhaeuser@hlrs.de: How to pin OpenMP-threads in hybrid jobs (naive approach pins 2 threads to _same_ core instead of two different), i.e.: how to do aprun -d?

• 2019-07-14, khabi/offenhaeuser@hlrs.de:

Modulefile best practices

• Set an environment variable to the root path of your installation (cf. e.g. MPI_ROOT in /usr/share/Modules/modulefiles/hmpt/2.19).
• Set not only CPATH but also respective variables used by PGI / Intel / etc. -> someone has to figure out the list of those variables, same w.r.t. (LD_)LiBRARY_PATH.
• Include your Name (finger does not work anymore), E-Mail and date of installation into the modulefile.
• It's possible to hold the modulefile(s) together with the actual installation in the respective directory and just create symlinks in /opt/hlrs/modulefiles/.
• Directory structure of /opt/hlrs/ shall be replicated in /opt/hlrs/unsupported-modulefiles/.
• In case of dependencies, load explicit versions instead of default one!

As an example a modulefile for package "foo" version 1.23 (within the category "performance") should look like:

#%Module1.0
BASE_DIR=/opt/hlrs/
CAT=performance
PACKAGE=Foo
VERSION=1.23

PACKAGE_ROOT=$BASE_DIR/$CAT/$PACKAGE/$VERSION

setenv PACKAGE_ROOT $PACKAGE_ROOT
setenv PACKAGE_VERSION $VERSION

prepend-path PATH                $PACKAGE_ROOT/bin
prepend-path LD_LIBRARY_PATH     $PACKAGE_ROOT/lib        # library search path at time of execution
prepend-path LIBRARY_PATH        $PACKAGE_ROOT/lib        # equivalent of "-L" for C, C++, and Fortran; works for gcc and clang
prepend-path CPATH               $PACKAGE_ROOT/include    # equivalent to "-I" for C and C++; works with gcc and clang
prepend-path CPLUS_INCLUDE_PATH  $PACKAGE_ROOT/include    # equivalent to "-I" only for C++ compiler; usually not needed as CPATH will do
#prepend-path C_INCLUDE_PATH      $PACKAGE_ROOT/include    # equivalent to "-I" only for C compiler; usually not needed as CPATH will do
###  not verified yet, seems to be intel specific: prepend-path FPATH               $PACKAGE_ROOT/include    # same for Fortran

Test cases best practices

Test cases will help to identify and determine the scaling/ performance behavior of the new system. Ideally, those test cases can be compared to other systems as well to get a full picture.

To do:

• Definition of a best practice guideline on how to set up a correct test case
  • Only measurement of time-stepping loop or equivalent excluding the initialization phase or cleanup
  • Well defined measure of computational progress (e.g. LUPS, DoF-UPS, Iterations/s or Flop/s)
  • Ideally, the test case is mostly automated with scripts and does also the evaluation on top with a meaningful result file