Module environment(Hawk)

Hints for users

A module environment is used on Hawk in order to allow for flexibly choosing software to be used out of a large set. In order to use a particular software, you first have to load the respective module. A set of default modules is already loaded after login.

A new implementation of the module command is used on Hawk in order to allow for improved user experience. However, the same sub-commands are available as known from former systems. The layout of outputs yet slightly differs.

Basic commands:

• show available modules:

module avail

or

module avail <substring to be searched for>

• show brief description for a given module:

module whatis <modulename>

• (un)load a module:

module (un)load <modulename>

• swap modules:

module swap <old module> <new module>

• show loaded modules:

module list

• show help for a specific module:

module help <modulename>

• search for keywords in module descriptions (e.g. "compiler", "numlib", "cae", etc.). Please have in mind that all modules will match which have the respective string in their whatis- or help-text. HLRS hence tries to set useful keywords surrounded by underscores (e.g. _compiler_, _mpi_, _performance_, etc.).

 module key <keyword>

Please have a look on here w.r.t. further functionality.

Module hierarchy: In order to work properly, applications built with a given compiler and MPI need to be linked with libraries built with the very same compiler and MPI. The module mechanism deployed on Hawk supports you in doing so by providing only those modules which are compatible with the loaded compiler and MPI. In case of swapping the compiler and/or MPI, available and loaded modules are also swapped in order to match with the new compiler/MPI. If there is no matching module, the respective module is set to the inactive state. Inactive modules will be reloaded as soon as a matching module becomes available again by swapping the compiler and/or MPI implementation.

In order to reveal dependencies, modules are assigned to one of four hierarchy levels:

• System modules provide some basic functionality of the system and can be ignored most of the time. The modules from this hierarchy level are always available.
• Core modules contains all the modules which are independent of the loaded compiler and MPI. In particular, it contains the compilers themselves. Also the modules from this hierarchy level are always available.
• Compiler-dependent modules contains all the modules compatible with the loaded compiler. The content of this level hence depends on the loaded compiler. Not all modules might be available with all compilers. However, we try to provide as much as possible. In particular, MPI implementations can be found here.
• MPI-dependent modules contains all the modules compatible with the loaded MPI implementation. The content of this level hence depends on the loaded MPI implementation. Again, not all modules might be available with all MPI implementations, but we try to provide as much as possible.

Due to the auto-swapping feature, the system allows for easily testing different compilers and MPI implementations. From time to time, you should do so in order to figure out the best compiler/MPI for your current application and/or use case. Please have in mind that changing the compiler/MPI might increase your performance and is almost for free!

module spider: In order to list all available modules without considering their dependencies, use module spider:

hpcoft28@login2:~$ module spider

------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
------------------------------------------------------------------------------------------------------------------------------------------------------
  adios2: adios2/2.5.0

  advisor: advisor/2019.4

  aocc: aocc/1.3.0, aocc/2.0.0, aocc/2.1.0

  boost: boost/1.70.0

  cgns: cgns/3.4.0-int32, cgns/3.4.0-int64

  cmake: cmake/3.15.2

  development/binutils: development/binutils/2.32

  fftw: fftw/3.3.8

  flex: flex/2.6.4

  forge: forge/19.1.2

  glm: glm/0.9.9.5

  gnu: gnu/9.1.0, gnu/9.2.0

  hdf5: hdf5/1.10.5

  inspector: inspector/2019.4

  intel: intel/18.0.2, intel/19.0.4, intel/19.1.0

  kahip: kahip/2.12-int32, kahip/2.12-int64

  metis: metis/5.1.0-int32-shared, metis/5.1.0-int32, metis/5.1.0-int64-shared, metis/5.1.0-int64

  mpt: mpt/2.20

  mumps: mumps/5.2.1-int32-shared, mumps/5.2.1-int32, mumps/5.2.1-int64-shared, mumps/5.2.1-int64

  netcdf: netcdf/4.5.0, netcdf/4.7.3

  netcdf-cxx4: netcdf-cxx4/4.3.1

  netcdf-fortran: netcdf-fortran/4.5.2

  openmpi: openmpi/4.0.2

  papi: papi/5.7.0

  parallel-netcdf: parallel-netcdf/1.11.2, parallel-netcdf/1.12.0

  parmetis: parmetis/4.0.3-int32-shared, parmetis/4.0.3-int32, parmetis/4.0.3-int64-shared, parmetis/4.0.3-int64

  parmgridgen: parmgridgen/1.0

  perfboost: perfboost

  perfcatcher: perfcatcher

  petsc: petsc/3.12.2-int64-shared, petsc/3.12.2-int64

  pgi: pgi/19.5, pgi/19.7

  python: python/3.8

  scalapack: scalapack/2.1.0-shared, scalapack/2.1.0

  scotch: scotch/6.0.9-int32-shared, scotch/6.0.9-int32, scotch/6.0.9-int64-shared, scotch/6.0.9-int64

  superlu: superlu/4.3, superlu/5.2.1

  system/site_names: system/site_names

  system/wrappers: system/wrappers/1.0

  system/ws: system/ws/f7ba528

  trilinos: trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0, trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0

  unsupported-modules: unsupported-modules

  uprof: uprof/3.1.35, uprof/3.2.228

  use.own: use.own

  vampir: vampir/9.7.0

  vampirserver: vampirserver/9.7.0

  vtune: vtune/2019.4

------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

------------------------------------------------------------------------------------------------------------------------------------------------------

 

hpcoft28@login2:~$ 

By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module:

hpcoft28@login2:~$ module spider fftw/3.3.8

------------------------------------------------------------------------------------------------------------------------------------------------------
  fftw: fftw/3.3.8
------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "fftw/3.3.8" module is available to load.

      gnu/9.1.0  mpt/2.20
      gnu/9.1.0  openmpi/4.0.2
 
    Help:
      FFTW is a C subroutine library for computing the discrete Fourier
      transform (DFT) in one or more dimensions, of arbitrary input size, and
      of both real and complex data (as well as of even/odd data, i.e. the
      discrete cosine/sine transforms or DCT/DST). We believe that FFTW, which
      is free software, should become the FFT library of choice for most
      applications.


 

hpcoft28@login2:~$ 

Remarks

• The modules are not arranged by means of folders (e.g. "compiler", "numlib", "cae", etc) anymore. Instead, it's possible to search by means of keywords. In order to show e.g. all the compilers, please do a:

module key compiler

• It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.

• If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). Actually the rule is more complicated. Please refer to here with respect to details.

• All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!

• We provide support for Bash only. In case of the module command not being defined (correctly), cf. here and here.

• Do an export LMOD_PAGER=None to switch off displaying multi-page outputs with a pager (e.g. less).

Using non-default revisions of the software stack

• From time to time, a new revision of the entire software stack is set to be default. If you like to use an older revision, please adhere to this method:

$> clearLmod
$> export MODULEPATH_ROOT=/opt/hlrs/spack/<revision you like to use>/modulefiles/linux-centos8-x86_64/
$> export MODULEPATH=$MODULEPATH_ROOT/gcc/8.3.1:/opt/hlrs/non-spack/system/modulefiles/
$> source /opt/hlrs/non-spack/system/profile.d/modules.sh
$> module load gcc mpt

• If doing so, please have in mind that support is provided for the default revision of the software stack only! Furthermore, old revisions will be deleted from time to time. Hence, port your application as soon as possible to the default revision!

Documentation

The entire official documentation can be found here. If you need further information, interesting sections might be:

• Advanced User Guide for Personal Modulefiles
• How to use a Software Module hierarchy
• Using the module spider command

Hints for modulefile providers

• Hawk uses Lmod instead of the traditional TCL based module implementation (called Tmod hereinafter). Lmod uses modulefiles written in Lua.

• Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. here and here)!

• If you like to use the extended capabilities of Lua modulefiles (cf. here), compare for the following resources with respect to the syntax to be used:
  • Writing modulefiles
  • Functions that can be used within modulefiles
  • Examples

• It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by

$LMOD_DIR/tcl2lua.tcl <TCL-modulefile>

• Please take care to add the suffix .lua to modulefiles written in Lua. Otherwise Lmod will ignore them.

• Note that by default, the umask for hpcoft* accounts is set to 0077. That means that files and folders created (either manually or when unpacking a tar file) cannot be read or entered for anyone but the user. For to ensure that others can also read your files/enter your folders, execute umask 022, which is the default for regular Linux users. In case you want to ensure that all files created by you can later be modified by other hpcoft* users, you can also set umask 002. In this case, everyone in your user group will by default have the same file permissions as yourself.

• Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.

• The module hierarchy mentioned above is implemented by means of adding additional paths to the MODULEPATH environment variable. Loading a given compiler e.g. adds a directory to MODULEPATH that holds all the modules compatible with that compiler. The same is true with respect to loading an MPI implementation.

• In case you install modules without Spack, please take care to place the respective modulefiles in the following locations:
  • modules built with system gcc or without a compiler: /opt/hlrs/non-spack/<revision, currently 009>/gcc/<version of system gcc, currently 8.5.0>/
  • modules built with another compiler: /opt/hlrs/non-spack/<revision, currently 009>/modulefiles/<compiler>/<compiler version>/
  • modules built with another compiler and MPI: /opt/hlrs/non-spack/<revision, currently 009>/modulefiles/<MPI implementation>/<MPI version>/<compiler>/<compiler version>

Remarks

• family tag : Modules can be tagged with a family name (cf. here). Lmod guarantees that only one member of every family is loaded at any time by unloading other members. This can and should be used for compiler as well as MPI modules.
• Load/unload semantics : There are several ways to load dependencies (cf. here) which differ in the precise semantics when loading/unloading a module. It's recommended to use depends_on("<modulename>").
• Default versions : To mark a version as default, create a symlink to the respective modulefile in the same directory as the modulefile and give it the name 'default' (cf. here).
• Spider Cache : If there is a system spider cache, please recreate it in case you added / modified a modulefile! You can do this by calling $(dirname ${LMOD_CMD})/update_lmod_system_cache_files (cf. here).

• PATH-like variables: It is important how duplicates in the PATH-like variables are handled. At the moment, LMOD_TMOD_PATH_RULE is set to yes in order to have the same behavior as the traditional module command (cf. here). Please have in mind that duplicates might be generated by:
  • Explicit loads of modules
  • Implicit loads due to modules loading further modules as prerequisites
  • Different modules inserting the same path

• 2-digit-rule: Lmod developers suggest to assume compatibility in between compilers and MPI stacks which differ in the third digit of the version number only.

Further nice features

• Module categories: module keyword allows for searching strings in the whatis-text (cf. here). By adding category names in the whatis-text, this feature can hence be used to implement module categories as known from Hazel Hen.

• Extensions: It’s possible to define “extensions”, i.e. some kind of pseudo-modules for stuff like numpy or scipy included in Python (but also e.g. FFTW included in AOCL, stuff included in Cray’s libsci, etc.). Those extensions will be listed by module avail and module spider in (almost) the same way as regular modules are listed. However, extensions can’t be loaded in the same way as regular modules, but Lmod tells the user how to do so after module load <extension-name> fails. With respect to details cf. here.

• Nag message: It’s possible to print messages when loading modules, e.g. to announce deprecation (cf. here and here).
  If your modulefile is written in Lua, this can also be accomplished by means of LmodMessage() (cf. here).

• Brief description of module: If adding whatis("Description: foobar"), the string “foobar” will be shown in the output of module spider below the module name / versions (cf. here).

• It’s possible to hide modulefiles and entire directories in the output of module avail and module spider by means of a leading dot in the file-/directory-name (cf. here).

• Hierarchy labels: In case you add a compiler or MPI implementation, take care to add the respective path to $(dirname $LMOD_CMD)/../init/hlrs/SitePackage.lua in order to prettily label the group of modules build with that compiler / MPI (cf. here).

• Debugging: Actions triggered by Lmod can be shown via module --trace <subcommand> (cf. here).

Documentation

The entire official documentation can be found here. If you need further information, interesting sections might be:

• An Introduction to Writing Modulefiles
• Lua Modulefile Functions
• Rules for PATH-like variables
• How to use a Software Module hierarchy
• Modulefile Examples from simple to complex
• Deprecating Modules
• Load Storms: Long load times or Fails to Load

Hints for system administrators

• Shared home: Some care has to be taken if running Lmod on multiple clusters with shared home filesystems (as is e.g. w.r.t. Hawk and Vulcan), cf. here.

System and site specific configuration

• MODULEPATH_ROOT and MODULEPATH are customized via $LMOD_CMD/../init/profile as follows:

#!/bin/sh
# -*- shell-script -*-
########################################################################
#  This is the system wide source file for setting up
#  modules:
#
########################################################################

if [ -z "${USER_IS_ROOT:-}" ]; then

  if [ -z "${MODULEPATH_ROOT:-}" ]; then
    export USER=${USER-${LOGNAME}}  # make sure $USER is set
    export LMOD_sys=`uname`

    # export MODULEPATH_ROOT="/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/modulefiles"
    export MODULEPATH_ROOT="/opt/hlrs/spack/current/modulefiles/linux-centos8-x86_64/"
    MODULEPATH_INIT="/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/lmod/lmod/init/.modulespath"
    # if [ -f ${MODULEPATH_INIT} ]; then
    #    for str in $(cat ${MODULEPATH_INIT} | sed 's/#.*$//'); do   # Allow end-of-line comments.
    #       for dir in $str; do
    #           if [ -d $dir ]; then
    #               export MODULEPATH=$(/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/lmod/lmod/libexec/a
    #           fi
    #   done
    #    done
    # else
    ⎸  # export MODULEPATH=$(/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/lmod/lmod/libexec/addto --appe
    ⎸  export MODULEPATH=$(/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/lmod/lmod/libexec/addto --append
    ⎸  # export MODULEPATH=$(/opt/hlrs/non-spack/system/lmod/lmod/8.2-gcc-8.3.1-45dqj25s/lmod/lmod/libexec/addto --appe
    # fi
...

• The entry points to the module hierarchy mentioned above (i.e. the paths always contained in MODULEPATH) are configured via $LMOD_CMD/../init/profile:28. Only the paths in this line should be contained in MODULEPATH.

• MODULEPATH_ROOT is set in $LMOD_CMD/../init/profile:16. Setting MODULEPATH via the "MODULEPATH_ROOT method" (cf. here) is required in order to allow for switching the software stack revision. Hence, usage of the file .modulespath (cf. here) is disabled by commenting out lines $LMOD_CMD/../init/profile:18-27,29-30.

• Definition of the module command is done in /opt/hlrs/non-spack/system/profile.d/modules.sh together with configuration of Lmod. Afterwards, the command is made available in shells via /opt/hlrs/non-spack/system/profile.d/zHWWenv.sh. Also the default compiler and MPI is loaded there. The respective files in /etc/profile.d/ are wrappers for those in /opt/hlrs/non-spack/system/profile.d/.

• Settings in /opt/hlrs/non-spack/system/profile.d/modules.sh furthermore link to $LMOD_CMD/../init/hlrs/SitePackage.lua which is used to translate entries of MODULEPATH to pretty names by means of the following code:

require("strict")
local hook = require("Hook")

local mapT =
{
   en_grouped = {
      ['linux%-centos8%-x86_64/mpt']              = "MPI-dependent modules",
      ['linux%-centos8%-x86_64/openmpi']          = "MPI-dependent modules",
      ['linux%-centos8%-x86_64/intel']            = "Compiler-dependent modules",
      ['linux%-centos8%-x86_64/clang']            = "Compiler-dependent modules",
      ['linux%-centos8%-x86_64/gcc/[^8]']         = "Compiler-dependent modules",
      ['linux%-centos8%-x86_64/gcc/8.3.1']        = "Core modules",
      ['/opt/hlrs/non%-spack/system/modulefiles'] = "System modules",
   },
   fr_grouped = {
      ['/Compilers$'] = "Compilateurs",
      ['/Core$']      = "Modules de base",
      ['/Common$']    = "Modules de base",
   },
}


function avail_hook(t)
   local availStyle = masterTbl().availStyle
   local styleT     = mapT[availStyle]
   if (not availStyle or availStyle == "system" or styleT == nil) then
      return
   end

   for k,v in pairs(t) do
      for pat,label in pairs(styleT) do
         if (k:find(pat)) then
            t[k] = "\27[1m"..label
            -- t[k] = "\27[1m"..label.."\27[0m   \27[2m("..k..")"
            break
         end
      end
   end
end


hook.register("avail",avail_hook)

Nice features

• Sticky modules: It’s possible to (almost) force particular packages to be always loaded (aka “sticky modules”, cf. here).

• Auto-completion: Setting LMOD_EXACT_MATCH=yes disables automatic selection of default versions, but prevents from selecting modules by specifying a prefix only (cf. here).

• Export module list: The spider tool can be used to export a list of available packages (e.g. for Wiki / Platform-Homepage, cf. here).

• It’s possible to track module usage (cf. here).

• It’s possible to use BLAS/LAPACK as an additional hierarchy layer (cf. here). We decided to use MKL as default for a start and evaluate performance of MKL vs. AOCL vs. PGI-version of BLAS/LAPACK vs. OpenBLAS later. Depending on the outcome, we might introduce an additional hierarchy layer in the future.

Documentation

The entire official documentation can be found here. If you need further information, interesting sections might be:

• Installing Lua and Lmod
• Configuring Lmod for your site
• The Interaction between Modules, MPI and Parallel Filesystems
• Lmod on Shared Home File Systems
• System Spider Cache
• Improving performance of Lmod