Hawk software installation

This pages outlines best practises for software installation on Hawk. It is primarily useful for HLRS staff. In addition, for HLRS users it might shed some light on the rational behind the peculiarities of the software installation.

Location for software installation

All software which is built specific for Hawk must be installed into the directory /opt/hlrs.

It is not allowed to use links between different platforms, i.e. /sw/<platform>.

Some software packages, mostly ISV codes, which are distributed as binary-only packages, might be installed into the directory /opt/general. However this should be an exception, not the norm.

Documentation of software installation

Documentation regarding the organisation and procedure of software installation should be in the staff wiki. This is intended for HLRS staff only.

Documentation describing the software installation, the layout of module hierarchies, individual software packages and their usage, etc, should be found in the Hawk platforms wiki.

Informing users about changes in the software stack

TBD. More discussion is needed on this.

E.g.

• MOTD
• Mailingliste schlank halten
• muss automatisch sein (siehe module-changes email)
• log der geladenen Module -> emails nur an User der jeweiligen Module verschicken?

Default versions of software packages

Each software package in the module hierarchy should have a sensible default version.

Module allows to set defaults through the .modulerc files. These can be in any directory in the module search path. The following example .modulerc sets the default version of the GCC module to 9.1.0:

#%Module1.0
#
#  May 17, 2018 Tünde Erdei <erdei@hlrs.de>
#
module-version  compiler/gnu/9.1.0     default

For the moment the default compiler is GCC. However, we should consider the AMD compiler AOCC as default.

The default MPI is mpt.

Modulefile best practices

• Set an environment variable to the root path of your installation (cf. e.g. MPI_ROOT in /usr/share/Modules/modulefiles/hmpt/2.19).
• Set not only CPATH but also respective variables used by PGI / Intel / etc. -> someone has to figure out the list of those variables, same w.r.t. (LD_)LiBRARY_PATH.
• Include your Name (finger does not work anymore), E-Mail and date of installation into the modulefile.
• It's possible to hold the modulefile(s) together with the actual installation in the respective directory and just create symlinks in /opt/hlrs/modulefiles/.
• Directory structure of /opt/hlrs/ shall be replicated in /opt/hlrs/unsupported-modulefiles/.
• In case of dependencies, load explicit versions instead of default one!

Note: Todos: change BASE_DIR to BASE_PRFIX (same as sit); add conflicts/requires/module load; add show_info

As an example a modulefile for package "foo" version 1.23 (within the category "performance") should look like:

#%Module1.0
#
# Change log:
#   Updated   12 Jul 2019, Christoph Niethammer <niethammer@hlrs.de>
#   Installed 08 Aug 2018, Jose Gracia <gracia@hlrs.de>
 

conflict module_that_should_not_be_loaded_at_the_same_time

module load module_required_by_this_one

BASE_DIR=/opt/hlrs/
CAT=performance
PACKAGE=Foo
VERSION=1.23

FOO_ROOT=$BASE_DIR/$CAT/$PACKAGE/$VERSION

setenv FOO_ROOT $FOO_ROOT
setenv FOO_VERSION $VERSION

prepend-path PATH                $FOO_ROOT/bin
prepend-path LD_LIBRARY_PATH     $FOO_ROOT/lib        # library search path at time of execution (i.e. in case of _dynamic_ linking)
prepend-path LIBRARY_PATH        $FOO_ROOT/lib        # equivalent of "-L" for C, C++, and Fortran
prepend-path CPATH               $FOO_ROOT/include    # equivalent to "-I" for C, C++ and Fortran
prepend-path CPLUS_INCLUDE_PATH  $FOO_ROOT/include    # equivalent to "-I" only for C++ compiler; usually not needed as CPATH will do
prepend-path C_INCLUDE_PATH      $FOO_ROOT/include    # equivalent to "-I" only for C compiler; usually not needed as CPATH will do
prepend-path MANPATH             $FOO_ROOT/share/man  # manpages

module-whatis "_brief_ description of what is provided by this module"

proc ModulesHelp {} {
    puts stderr "First line of detailed description\n"
    puts stderr "Second line of detailed description\n"
    puts stderr "Third line of detailed description\n"
}

Hierarchy of modules

On Vulcan and Hazel Hen we have various module directories like tools, utils, misc. However, it is not very clear what should go here; at the end anything is a tool. I would therefore propose to be more specific.

Proposal for module hierarchy (please extend)

development/    # development tools
    # svn, git, binutils, cmake 

mpi/             # nobody will suspect MPI under mpt/ 
    # mpt, hmpt, ....

compiler/
    # gcc, oacc, intel, pgi, ...

numlib/         # numerical libraries
    # mkl, trillinos, ...

debugger/       # debugging tools
    # forge, ...

performance/    # performance analysis tools
    # vampir, extrae, scalasca, inspector, advisor, darshan, ...

visualization/  # data visualization tools
    # paraview, ...

python/
    # 3.X; do we need/want 2.7?

What to do with libraries which are not necessarily "numlib", e.g.

boost/ -> libraries/boost
hdf5/  -> libraries/hdf5? io/hdf5?

Libraries which are actually some kind of programming model:

gpi2    -> libraries/gpi2
tbb     -> libraries/tbb

What to do with software for projects? Such as

tools/hidalgo/fenics_hpc/3dairq
prace/prace

Put them in directories which are readable only for a certain group?

Topics for Software Installation Meeting

• Installationsverzeichnis: /opt/hlrs; Platform unabhängig (Ausnahme): /sw/general; Links zwischen Platformen /sw/* dürfen nicht mehr verwendet werden
• Dokumentation der Installation: für User: platforms-Wiki; für Installer: staff-Wiki
• how to inform users about software updates in the future: keine Verbindung von platform ins staff wiki

MOTD?; Mailingliste schlank halten; muss automatisch sein (siehe module-changes email); wer lädt wann welche module -> relevante emails verschicken

• hands-on sit (software installation tool)
• template for modulefiles; see #Modulefile best practices
• hierarchy of modulefiles #Hierarchy of modules
• default compiler / MPI / others
• how to deal with software from base linux which should not be visible on cluster, i.e. system compiler (gcc 4.8.5)
• install location of python bindings