Module environment(Hawk)

Hints for users

A module environment is used on Hawk in order to allow for flexibly choosing software to be used out of a large set. In order to use a particular software, you first have to load the respective module. A set of default modules is already loaded after login.

A new implementation of the module command is used on Hawk in order to allow for improved user experience. However, the same sub-commands are available as known from former systems. The layout of outputs yet slightly differs. Furthermore, not all available modules are shown right after login (cf. below).

In order to use this new implementation of the module command, you have to generate a file ~/.lmod:

touch ~/.lmod

In order to use the old implementation, remove this file and have in mind that not all of the commands listed below are possible. Furthermore, output and behavior will differ.

Basic commands:

• show available modules:

module avail

or

module avail <substring to be searched for>

• show brief description for a given module:

module whatis <modulename>

• (un)load a module:

module (un)load <modulename>

• swap modules:

module swap <old module> <new module>

• show loaded modules:

module list

• show help for a specific module:

module help <modulename>

• search for keywords (e.g. “compiler”, “debugger”, etc.) in module descriptions:

 module key <keyword>

Please have a look on here w.r.t. further functionality.

Module categories: In order to work properly, applications built with a given compiler and MPI need to be linked with libraries built with the very same compiler and MPI. The module mechanism deployed on Hawk supports you in doing so by providing only those modules built with the loaded compiler and MPI. In case of swapping the compiler and/or MPI, available and loaded modules are also swapped in order to match with the new compiler/MPI.

For this purpose, modules are assigned to one of the categories

• System modules provide some basic functionality of the system and can be ignored most of the time. The modules from this category are always available.
• Core modules contains all the modules which are independent of the loaded compiler and MPI. In particular, it contains the compilers themselves. Also the modules from this category are always available.
• Compiler-dependent modules contains all the modules compatible with the loaded compiler. The content of this category hence depends on the loaded compiler. Not all modules might be available with all compilers. However, we try to provide as much as possible. In particular, MPI implementations can be found in this category.
• MPI-dependent modules contains all the modules compatible with the loaded MPI implementation. The content of this category hence depends on the loaded MPI implementation. Again, not all modules might be available with all MPI implementations, but we try to provide as much as possible.

module spider: In order to list all available modules without considering their dependencies, use module spider:

hpcoft28@login2:~$ module spider

------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
------------------------------------------------------------------------------------------------------------------------------------------------------
  adios2: adios2/2.5.0

  advisor: advisor/2019.4

  aocc: aocc/1.3.0, aocc/2.0.0, aocc/2.1.0

  boost: boost/1.70.0

  cgns: cgns/3.4.0-int32, cgns/3.4.0-int64

  cmake: cmake/3.15.2

  development/binutils: development/binutils/2.32

  fftw: fftw/3.3.8

  flex: flex/2.6.4

  forge: forge/19.1.2

  glm: glm/0.9.9.5

  gnu: gnu/9.1.0, gnu/9.2.0

  hdf5: hdf5/1.10.5

  inspector: inspector/2019.4

  intel: intel/18.0.2, intel/19.0.4, intel/19.1.0

  kahip: kahip/2.12-int32, kahip/2.12-int64

  metis: metis/5.1.0-int32-shared, metis/5.1.0-int32, metis/5.1.0-int64-shared, metis/5.1.0-int64

  mpt: mpt/2.20

  mumps: mumps/5.2.1-int32-shared, mumps/5.2.1-int32, mumps/5.2.1-int64-shared, mumps/5.2.1-int64

  netcdf: netcdf/4.5.0, netcdf/4.7.3

  netcdf-cxx4: netcdf-cxx4/4.3.1

  netcdf-fortran: netcdf-fortran/4.5.2

  openmpi: openmpi/4.0.2

  papi: papi/5.7.0

  parallel-netcdf: parallel-netcdf/1.11.2, parallel-netcdf/1.12.0

  parmetis: parmetis/4.0.3-int32-shared, parmetis/4.0.3-int32, parmetis/4.0.3-int64-shared, parmetis/4.0.3-int64

  parmgridgen: parmgridgen/1.0

  perfboost: perfboost

  perfcatcher: perfcatcher

  petsc: petsc/3.12.2-int64-shared, petsc/3.12.2-int64

  pgi: pgi/19.5, pgi/19.7

  python: python/3.8

  scalapack: scalapack/2.1.0-shared, scalapack/2.1.0

  scotch: scotch/6.0.9-int32-shared, scotch/6.0.9-int32, scotch/6.0.9-int64-shared, scotch/6.0.9-int64

  superlu: superlu/4.3, superlu/5.2.1

  system/site_names: system/site_names

  system/wrappers: system/wrappers/1.0

  system/ws: system/ws/f7ba528

  trilinos: trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0, trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0

  unsupported-modules: unsupported-modules

  uprof: uprof/3.1.35, uprof/3.2.228

  use.own: use.own

  vampir: vampir/9.7.0

  vampirserver: vampirserver/9.7.0

  vtune: vtune/2019.4

------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

------------------------------------------------------------------------------------------------------------------------------------------------------

 

hpcoft28@login2:~$ 

By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module:

hpcoft28@login2:~$ module spider fftw/3.3.8

------------------------------------------------------------------------------------------------------------------------------------------------------
  fftw: fftw/3.3.8
------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "fftw/3.3.8" module is available to load.

      gnu/9.1.0  mpt/2.20
      gnu/9.1.0  openmpi/4.0.2
 
    Help:
      FFTW is a C subroutine library for computing the discrete Fourier
      transform (DFT) in one or more dimensions, of arbitrary input size, and
      of both real and complex data (as well as of even/odd data, i.e. the
      discrete cosine/sine transforms or DCT/DST). We believe that FFTW, which
      is free software, should become the FFT library of choice for most
      applications.


 

hpcoft28@login2:~$ 

Environment changes: In case a user swaps the loaded compiler or MPI implementation, the module system tries to automatically reload matching modules from the compiler- and MPI-dependent sections. If there is no matching module, the respective module is set to the inactive state.
However, HLRS aspires to provide modules for as many compiler/MPI combinations as possible. From time to time, you should make use of this by profiling your code for different compiler/MPI combinations. Using different compilers and MPI implementations might significantly reduce the runtime of your code and is almost for free!

Details

• It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.

• If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). In fact, the rule is more complicated. Please refer to here with respect to details.

• All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!

• We provide support for Bash only. In case of the module command not being defined (correctly), cf. here and here.

• Do an export LMOD_PAGER=None to turn off displaying multi-page outputs with a pager (e.g. less).

Hints for modulefile providers

• Instead of the traditional one (Link, to be called Tmod hereinafter) Hawk uses a new implementation of the module command called Lmod (Link). While Tmod uses modulefiles written in TCL, Lmod uses modulefiles written in Lua.

• Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. here and here)!

• If writing a new modulefile, you should nevertheless prefer Lua over TCL. Compare for the following resources with respect to the syntax to be used:
  • Writing modulefiles
  • Functions that can be used within modulefiles
  • Examples

• It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by

$(dirname ${LMOD_CMD})/tcl2lua.tcl <TCL-modulefile>

• Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.

• In case you install modules without Spack, please take care to place the respective modulefiles in the following locations:
  • modules without dependencies: /opt/hlrs/modulefiles_lmod/Core/
  • modules depending on compiler: /opt/hlrs/modulefiles_lmod/<compiler>/<compiler version>/
  • modules depending on compiler and MPI: /opt/hlrs/modulefiles_lmod/<MPI implementation>/<MPI version>-<hash>/<compiler>/<compiler version>
  • The above directories should already exist, so you don't have to care about the correct <hash>.

Details

• Spider Cache : If there is a system spider cache, please recreate it in case you added / modified a modulefile! You can do this by calling $(dirname ${LMOD_CMD})/update_lmod_system_cache_files (cf. here).

• PATH-like variables: It is important how duplicates in the PATH-like variables are handled. At the moment, LMOD_TMOD_PATH_RULE is set to yes in order to have the same behavior as the traditional module command (cf. here). Please have in mind that duplicates might be generated by:
  • Explicit loads of modules
  • Implicit loads due to modules loading further modules as prerequisites
  • Different modules inserting the same path

• 2-digit-rule: Lmod developers suggest to assume compatibility in between compilers and MPI stacks which differ in the third digit of the version number only.

• Shared home: Some care has to be taken if running Lmod on multiple clusters with shared home filesystems (as is e.g. w.r.t. Hawk and Vulcan), cf. here.

In case of problems, contact Björn Dick <dick@hlrs.de>.

Further nice features

• Extensions: It’s possible to define “extensions”, i.e. some kind of pseudo-modules for stuff like numpy or scipy included in Python (but also e.g. FFTW included in AOCL, stuff included in Cray’s libsci, etc.). Those extensions will be listed by module avail and module spider in (almost) the same way as regular modules are listed. However, extensions can’t be loaded in the same way as regular modules, but Lmod tells the user how to do so after module load <extension-name> fails. With respect to details cf. here.

• Nag message: It’s possible to print messages when loading modules, e.g. to announce deprecation (cf. here and here).

• Brief description of module: If adding whatis("Description: foobar"), the string “foobar” will be shown in the output of module spider below the module name / versions (cf. here).

• It’s possible to hide modulefiles and entire directories in the output of module avail and module spider by means of a leading dot in the file-/directory-name (cf. here).

• Hierarchy labels: In case you add a compiler or MPI implementation, take care to add the respective path to $(dirname $LMOD_CMD)/../init/hlrs/SitePackage.lua in order to prettily label the group of modules build with that compiler / MPI (cf. here).

• Debugging: Actions triggered by Lmod can be shown via module --trace <subcommand> (cf. here).

Hints for system administrators

• Sticky modules: It’s possible to (almost) force particular packages to be always loaded (aka “sticky modules”, cf. here).

• Auto-completion: Setting LMOD_EXACT_MATCH=yes disables automatic selection of default versions, but prevents from selecting modules by specifying a prefix only (cf. here).

• Module categories: module keyword allows for searching strings in the whatis-text (cf. here). By adding category names in the whatis-text, this feature can hence be used to implement module categories as known from Hazel Hen.

• Export module list: The spider tool can be used to export a list of available packages (e.g. for Wiki / Platform-Homepage, cf. here).

• It’s possible to track module usage (cf. here).

• It’s possible to use BLAS/LAPACK as an additional hierarchy layer (cf. here). We decided to use MKL as default for a start and evaluate performance of MKL vs. AOCL vs. PGI-version of BLAS/LAPACK vs. OpenBLAS later. Depending on the outcome, we might introduce an additional hierarchy layer in the future.