- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -

Difference between revisions of "Module environment(Hawk)"

From HLRS Platforms
(Hints for modulefile providers)
(Hints for modulefile providers)
(49 intermediate revisions by the same user not shown)
Line 10: Line 10:
  
 
'''Basic commands:'''
 
'''Basic commands:'''
* show available modules: module avail or module avail <substring to be searched for>
+
 
* show brief description for a given module: module whatis <modulename>
+
* show available modules:
* (un)load a module: module (un)load <modulename>
+
  <pre>module avail</pre> or
* swap modules: module swap <old module> <new module>
+
  <pre>module avail <substring to be searched for></pre>
* show loaded modules: module list
+
 
* show help for a specific module: module help <modulename>
+
* show brief description for a given module:
* search for keywords (e.g. “compiler”, “debugger”, etc.) in module descriptions: module keyword <keyword>
+
  <pre>module whatis <modulename></pre>
 +
 
 +
* (un)load a module:
 +
  <pre>module (un)load <modulename></pre>
 +
 
 +
* swap modules:
 +
  <pre>module swap <old module> <new module></pre>
 +
 
 +
* show loaded modules:
 +
  <pre>module list</pre>
 +
 
 +
* show help for a specific module:
 +
  <pre>module help <modulename></pre>
 +
 
 +
* search for keywords (e.g. “compiler”, “debugger”, etc.) in module descriptions:
 +
  <pre> module key <keyword></pre>
  
 
<br>
 
<br>
Line 28: Line 43:
 
<br>
 
<br>
  
Right after login, you can load so-called system and core modules only. After loading a compiler, module avail shows also the modules depending on that compiler. In this set, you can - in particular - find MPI implementations. After loading one of those, module avail also shows the modules depending on the loaded compiler as well as MPI. <br>
+
'''''Workflow''''': Right after login, you can load so-called system and core modules only:
In order to list ''all'' available modules ''without'' considering their dependencies, use module spider. By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module.
+
<br>
 +
<pre>
 +
hpcoft28@login2:~$ module avail
 +
 
 +
---------------------------------------------- Core Modules  (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
 +
  advisor/2019.4        development/binutils/2.32        intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
 +
  aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
 +
  aocc/2.0.0            gnu/9.1.0                        intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0
 +
  aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
 +
  cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
 +
 
 +
---------------------------------------------- System Modules  (/opt/hlrs/system/modulefiles) -----------------------------------------------
 +
  system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own
 +
 
 +
  Where:
 +
  L:  Module is loaded
 +
  D:  Default Module
 +
 
 +
Use "module spider" to find all possible modules and extensions.
 +
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
 +
 
 +
 
 +
hpcoft28@login2:~$
 +
</pre>
 +
<br>
 +
After loading a compiler, module avail shows also the modules depending on that compiler:
 +
<br>
 +
<pre>
 +
hpcoft28@login2:~$ module load gnu/9.1.0
 +
hpcoft28@login2:~$ module avail
 +
 
 +
-------- Compiler-dependent modules  (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
 +
  flex/2.6.4    metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20        superlu/4.3
 +
  glm/0.9.9.5    metis/5.1.0-int32          metis/5.1.0-int64        (D)    openmpi/4.0.2    superlu/5.2.1 (D)
 +
 
 +
---------------------------------------------- Core Modules  (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
 +
  advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
 +
  aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
 +
  aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver
 +
  aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
 +
  cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
 +
 
 +
---------------------------------------------- System Modules  (/opt/hlrs/system/modulefiles) -----------------------------------------------
 +
  system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own
 +
 
 +
  Where:
 +
  L:  Module is loaded
 +
  D:  Default Module
 +
 
 +
Use "module spider" to find all possible modules and extensions.
 +
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
 +
 
 +
 
 +
hpcoft28@login2:~$
 +
</pre>
 +
<br>
 +
In this set, you can - in particular - find MPI implementations. After loading one of those, module avail also shows the modules depending on the loaded compiler as well as MPI:
 +
<br>
 +
<pre>
 +
hpcoft28@login2:~$ module load mpt/2.20
 +
hpcoft28@login2:~$ module avail
 +
 
 +
-- MPI-dependent modules  (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/mpt/2.20-44w73ly/gcc/9.1.0) --
 +
  adios2/2.5.0                    mumps/5.2.1-int64          (D)    parmgridgen/1.0
 +
  boost/1.70.0                    netcdf-cxx4/4.3.1                  petsc/3.12.2-int64-shared
 +
  cgns/3.4.0-int32                netcdf-fortran/4.5.2              petsc/3.12.2-int64                                          (D)
 +
  cgns/3.4.0-int64        (D)    netcdf/4.5.0                      scalapack/2.1.0-shared
 +
  fftw/3.3.8                      netcdf/4.7.3                (D)    scalapack/2.1.0                                            (D)
 +
  hdf5/1.10.5                    parallel-netcdf/1.11.2            scotch/6.0.9-int32-shared
 +
  kahip/2.12-int32                parallel-netcdf/1.12.0      (D)    scotch/6.0.9-int32
 +
  kahip/2.12-int64        (D)    parmetis/4.0.3-int32-shared        scotch/6.0.9-int64-shared
 +
  mumps/5.2.1-int32-shared        parmetis/4.0.3-int32              scotch/6.0.9-int64                                          (D)
 +
  mumps/5.2.1-int32              parmetis/4.0.3-int64-shared        trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0
 +
  mumps/5.2.1-int64-shared        parmetis/4.0.3-int64        (D)    trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0 (D)
 +
 
 +
-------- Compiler-dependent modules  (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
 +
  flex/2.6.4    metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20      (L)    superlu/4.3
 +
  glm/0.9.9.5    metis/5.1.0-int32          metis/5.1.0-int64        (D)    openmpi/4.0.2        superlu/5.2.1 (D)
 +
 
 +
---------------------------------------------- Core Modules  (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
 +
  advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
 +
  aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
 +
  aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0
 +
  aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
 +
  cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
 +
 
 +
---------------------------------------------- System Modules  (/opt/hlrs/system/modulefiles) -----------------------------------------------
 +
  system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own
 +
 
 +
  Where:
 +
  L:  Module is loaded
 +
  D:  Default Module
 +
 
 +
Use "module spider" to find all possible modules and extensions.
 +
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
 +
 
 +
 
 +
hpcoft28@login2:~$
 +
</pre>
 +
<br>
 +
<br>
 +
 
 +
'''''module spider''''':
 +
In order to list ''all'' available modules ''without'' considering their dependencies, use module spider:
 +
<br>
 +
<pre>
 +
hpcoft28@login2:~$ module spider
 +
 
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
The following is a list of the modules and extensions currently available:
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
  adios2: adios2/2.5.0
 +
 
 +
  advisor: advisor/2019.4
 +
 
 +
  aocc: aocc/1.3.0, aocc/2.0.0, aocc/2.1.0
 +
 
 +
  boost: boost/1.70.0
 +
 
 +
  cgns: cgns/3.4.0-int32, cgns/3.4.0-int64
 +
 
 +
  cmake: cmake/3.15.2
 +
 
 +
  development/binutils: development/binutils/2.32
 +
 
 +
  fftw: fftw/3.3.8
 +
 
 +
  flex: flex/2.6.4
 +
 
 +
  forge: forge/19.1.2
 +
 
 +
  glm: glm/0.9.9.5
 +
 
 +
  gnu: gnu/9.1.0, gnu/9.2.0
 +
 
 +
  hdf5: hdf5/1.10.5
 +
 
 +
  inspector: inspector/2019.4
 +
 
 +
  intel: intel/18.0.2, intel/19.0.4, intel/19.1.0
 +
 
 +
  kahip: kahip/2.12-int32, kahip/2.12-int64
 +
 
 +
  metis: metis/5.1.0-int32-shared, metis/5.1.0-int32, metis/5.1.0-int64-shared, metis/5.1.0-int64
 +
 
 +
  mpt: mpt/2.20
 +
 
 +
  mumps: mumps/5.2.1-int32-shared, mumps/5.2.1-int32, mumps/5.2.1-int64-shared, mumps/5.2.1-int64
 +
 
 +
  netcdf: netcdf/4.5.0, netcdf/4.7.3
 +
 
 +
  netcdf-cxx4: netcdf-cxx4/4.3.1
 +
 
 +
  netcdf-fortran: netcdf-fortran/4.5.2
 +
 
 +
  openmpi: openmpi/4.0.2
 +
 
 +
  papi: papi/5.7.0
 +
 
 +
  parallel-netcdf: parallel-netcdf/1.11.2, parallel-netcdf/1.12.0
 +
 
 +
  parmetis: parmetis/4.0.3-int32-shared, parmetis/4.0.3-int32, parmetis/4.0.3-int64-shared, parmetis/4.0.3-int64
 +
 
 +
  parmgridgen: parmgridgen/1.0
 +
 
 +
  perfboost: perfboost
 +
 
 +
  perfcatcher: perfcatcher
 +
 
 +
  petsc: petsc/3.12.2-int64-shared, petsc/3.12.2-int64
 +
 
 +
  pgi: pgi/19.5, pgi/19.7
 +
 
 +
  python: python/3.8
 +
 
 +
  scalapack: scalapack/2.1.0-shared, scalapack/2.1.0
 +
 
 +
  scotch: scotch/6.0.9-int32-shared, scotch/6.0.9-int32, scotch/6.0.9-int64-shared, scotch/6.0.9-int64
 +
 
 +
  superlu: superlu/4.3, superlu/5.2.1
 +
 
 +
  system/site_names: system/site_names
 +
 
 +
  system/wrappers: system/wrappers/1.0
 +
 
 +
  system/ws: system/ws/f7ba528
 +
 
 +
  trilinos: trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0, trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0
 +
 
 +
  unsupported-modules: unsupported-modules
 +
 
 +
  uprof: uprof/3.1.35, uprof/3.2.228
 +
 
 +
  use.own: use.own
 +
 
 +
  vampir: vampir/9.7.0
 +
 
 +
  vampirserver: vampirserver/9.7.0
 +
 
 +
  vtune: vtune/2019.4
 +
 
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
 
 +
To learn more about a package execute:
 +
 
 +
  $ module spider Foo
 +
 
 +
where "Foo" is the name of a module.
 +
 
 +
To find detailed information about a particular package you
 +
must specify the version if there is more than one version:
 +
 
 +
  $ module spider Foo/11.1
 +
 
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
 
 +
 +
 
 +
hpcoft28@login2:~$
 +
</pre>
 +
<br>
 +
By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module:
 +
<br>
 +
<pre>
 +
hpcoft28@login2:~$ module spider fftw/3.3.8
 +
 
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
  fftw: fftw/3.3.8
 +
------------------------------------------------------------------------------------------------------------------------------------------------------
 +
 
 +
    You will need to load all module(s) on any one of the lines below before the "fftw/3.3.8" module is available to load.
 +
 
 +
      gnu/9.1.0  mpt/2.20
 +
      gnu/9.1.0  openmpi/4.0.2
 +
 +
    Help:
 +
      FFTW is a C subroutine library for computing the discrete Fourier
 +
      transform (DFT) in one or more dimensions, of arbitrary input size, and
 +
      of both real and complex data (as well as of even/odd data, i.e. the
 +
      discrete cosine/sine transforms or DCT/DST). We believe that FFTW, which
 +
      is free software, should become the FFT library of choice for most
 +
      applications.
 +
 
 +
 
 +
 +
 
 +
hpcoft28@login2:~$
 +
</pre>
 +
 
 +
<br>
 +
<br>
 +
 
 +
'''''Environment changes''''':
 +
In case a user swaps the loaded compiler or MPI implementation, the module system tries to automatically reload matching modules from the compiler- and MPI-dependent sections. If there is no matching module, the respective module is set to the inactive state.<br>
 +
However, HLRS aspires to provide modules for as many compiler/MPI combinations as possible. From time to time, you should make use of this by profiling your code for different compiler/MPI combinations. <font color="red">Using different compilers and MPI implementations might significantly reduce the runtime of your code and is almost for free!</font>
  
 
<br>
 
<br>
  
'''Please have in mind:'''
+
=== Details ===
  
 
* It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.
 
* It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.
 +
 
* If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). In fact, the rule is more complicated. Please refer to [https://lmod.readthedocs.io/en/latest/060_locating.html here] with respect to details.
 
* If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). In fact, the rule is more complicated. Please refer to [https://lmod.readthedocs.io/en/latest/060_locating.html here] with respect to details.
 +
 
* All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!
 
* All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!
 +
 
* We provide support for Bash only. In case of the module command not being defined (correctly), cf. [https://lmod.readthedocs.io/en/latest/030_installing.html#integrating-module-into-users-shells here] and [https://lmod.readthedocs.io/en/latest/030_installing.html#issues-with-bash here].
 
* We provide support for Bash only. In case of the module command not being defined (correctly), cf. [https://lmod.readthedocs.io/en/latest/030_installing.html#integrating-module-into-users-shells here] and [https://lmod.readthedocs.io/en/latest/030_installing.html#issues-with-bash here].
 +
 
* Do an export LMOD_PAGER=None to turn off displaying multi-page outputs with a pager (e.g. less).
 
* Do an export LMOD_PAGER=None to turn off displaying multi-page outputs with a pager (e.g. less).
  
 
+
<br>
 
+
<br>
 
+
<br>
  
 
== Hints for modulefile providers ==
 
== Hints for modulefile providers ==
  
 
* Instead of the traditional one ([http://modules.sourceforge.net/ Link], to be called Tmod hereinafter) Hawk uses a new implementation of the module command called Lmod ([https://lmod.readthedocs.io/en/latest/ Link]). While Tmod uses modulefiles written in TCL, Lmod uses modulefiles written in Lua.
 
* Instead of the traditional one ([http://modules.sourceforge.net/ Link], to be called Tmod hereinafter) Hawk uses a new implementation of the module command called Lmod ([https://lmod.readthedocs.io/en/latest/ Link]). While Tmod uses modulefiles written in TCL, Lmod uses modulefiles written in Lua.
 +
 
* Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. [https://lmod.readthedocs.io/en/latest/073_tmod_to_lmod.html#converting-from-tcl-c-environment-modules-to-lmod here] and [https://lmod.readthedocs.io/en/latest/095_tcl2lua.html#tcl2lua-label here])!
 
* Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. [https://lmod.readthedocs.io/en/latest/073_tmod_to_lmod.html#converting-from-tcl-c-environment-modules-to-lmod here] and [https://lmod.readthedocs.io/en/latest/095_tcl2lua.html#tcl2lua-label here])!
 +
 
* If writing a new modulefile, you should nevertheless prefer Lua over TCL. Compare for the following resources with respect to the syntax to be used:
 
* If writing a new modulefile, you should nevertheless prefer Lua over TCL. Compare for the following resources with respect to the syntax to be used:
 
** [https://lmod.readthedocs.io/en/latest/015_writing_modules.html Writing modulefiles]
 
** [https://lmod.readthedocs.io/en/latest/015_writing_modules.html Writing modulefiles]
 
** [https://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html Functions that can be used within modulefiles]
 
** [https://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html Functions that can be used within modulefiles]
 
** [https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html Examples]
 
** [https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html Examples]
* It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by calling $(dirname ${LMOD_CMD})/tcl2lua.tcl.
+
 
 +
* It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by
 +
<pre>$(dirname ${LMOD_CMD})/tcl2lua.tcl <TCL-modulefile></pre>
 +
 
 
* Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.
 
* Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.
 +
 +
* In case you install modules without Spack, please take care to place the respective modulefiles in the following locations:
 +
** modules without dependencies: /opt/hlrs/modulefiles_lmod/Core/
 +
** modules depending on compiler: /opt/hlrs/modulefiles_lmod/<compiler>/<compiler version>/
 +
** modules depending on compiler and MPI: /opt/hlrs/modulefiles_lmod/<MPI implementation>/<MPI version>-<hash>/<compiler>/<compiler version>
 +
** The above directories should already exist, so you don't have to care about the correct <hash>.
  
 
<br>
 
<br>
  
'''Please also have in mind:'''
+
=== Details ===
 +
* ''''' Spider Cache ''''': If there is a system spider cache, please recreate it in case you added / modified a modulefile! You can do this by calling $(dirname ${LMOD_CMD})/update_lmod_system_cache_files (cf. [https://lmod.readthedocs.io/en/latest/130_spider_cache.html#system-spider-cache here]).
 +
 
 +
* '''''PATH-like variables''''': It is important how duplicates in the PATH-like variables are handled. At the moment, LMOD_TMOD_PATH_RULE is set to yes in order to have the same behavior as the traditional module command (cf. [https://lmod.readthedocs.io/en/latest/077_ref_counting.html here]). Please have in mind that duplicates might be generated by:
 +
** Explicit loads of modules
 +
** Implicit loads due to modules loading further modules as prerequisites
 +
** Different modules inserting the same path
 +
 
 +
* '''''2-digit-rule''''': Lmod developers suggest to assume compatibility in between compilers and MPI stacks which differ in the third digit of the version number only.
 +
 
 +
* '''''Shared home''''': Some care has to be taken if running Lmod on multiple clusters with shared home filesystems (as is e.g. w.r.t. Hawk and Vulcan), cf. [https://lmod.readthedocs.io/en/latest/120_shared_home_directories.html here].
 +
 
 +
In case of problems, contact Björn Dick <dick@hlrs.de>.
 +
 
 +
<br>
 +
 
 +
=== Further nice features ===
 +
 
 +
* '''''Extensions''''': It’s possible to define “extensions”, i.e. some kind of pseudo-modules for stuff like numpy or scipy included in Python (but also e.g. FFTW included in AOCL, stuff included in Cray’s libsci, etc.). Those extensions will be listed by module avail and module spider in (almost) the same way as regular modules are listed. However, extensions can’t be loaded in the same way as regular modules, but Lmod tells the user how to do so after module load <extension-name> fails. With respect to details cf. [https://lmod.readthedocs.io/en/latest/330_extensions.html here].
 +
 
 +
* '''''Nag message''''': It’s possible to print messages when loading modules, e.g. to announce deprecation (cf. [https://lmod.readthedocs.io/en/latest/140_deprecating_modules.html here] and [https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html#environment-variables-only here]).
 +
 
 +
* '''''Brief description of module''''': If adding whatis("Description: foobar"), the string “foobar” will be shown in the output of module spider below the module name / versions (cf. [https://lmod.readthedocs.io/en/latest/135_module_spider.html#level-0-module-spider here]).
 +
 
 +
* It’s possible to '''''hide modulefiles''''' and entire directories in the output of module avail and module spider by means of a leading dot in the file-/directory-name (cf. [https://lmod.readthedocs.io/en/latest/055_module_names.html#special-names-for-modulefiles here]).
 +
 
 +
* '''''Hierarchy labels''''': In case you add a compiler or MPI implementation, take care to add the respective path to $(dirname $LMOD_CMD)/../init/hlrs/SitePackage.lua in order to prettily label the group of modules build with that compiler / MPI (cf. [https://lmod.readthedocs.io/en/latest/200_avail_custom.html here]).
 +
 
 +
* '''''Debugging''''': Actions triggered by Lmod can be shown via module --trace <subcommand> (cf. [https://lmod.readthedocs.io/en/latest/220_tracing.html#tracing-lmod here]).
 +
 
 +
<br>
 +
<br>
 +
<br>
 +
 
 +
== Hints for system administrators ==
 +
 
 +
* '''''Sticky modules''''': It’s possible to (almost) force particular packages to be always loaded (aka “sticky modules”, cf. [https://lmod.readthedocs.io/en/latest/240_sticky_modules.html here]).
 +
 
 +
* '''''Auto-completion''''': Setting LMOD_EXACT_MATCH=yes disables automatic selection of default versions, but prevents from selecting modules by specifying a prefix only (cf. [https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html#configuration-time-settings-that-can-be-overridden-by-env-vars here]).
 +
 
 +
* '''''Module categories''''': module keyword allows for searching strings in the whatis-text (cf. [https://lmod.readthedocs.io/en/latest/010_user.html#user-s-tour-of-the-module-command here]). By adding category names in the whatis-text, this feature can hence be used to implement module categories as known from Hazel Hen.
 +
 
 +
* '''''Export module list''''': The spider tool can be used to export a list of available packages (e.g. for Wiki / Platform-Homepage, cf. [https://lmod.readthedocs.io/en/latest/136_spider.html#software-page-generation here]).
 +
 
 +
* It’s possible to '''''track module usage''''' (cf. [https://lmod.readthedocs.io/en/latest/300_tracking_module_usage.html here]).
 +
 
 +
* It’s possible to use '''''BLAS/LAPACK as an additional hierarchy layer''''' (cf. [https://spack.readthedocs.io/en/latest/module_file_support.html#customize-the-naming-scheme here]). We decided to use MKL as default for a start and evaluate performance of MKL vs. AOCL vs. PGI-version of BLAS/LAPACK vs. OpenBLAS later. Depending on the outcome, we might introduce an additional hierarchy layer in the future.

Revision as of 17:54, 15 January 2020

Hints for users

A module environment is used on Hawk in order to allow for flexibly choosing software to be used out of a large set. In order to use a particular software, you first have to load the respective module. A set of default modules is already loaded after login.


A new implementation of the module command is used on Hawk in order to allow for improved user experience. However, the same sub-commands are available as known from former systems. The layout of outputs yet slightly differs. Furthermore, not all available modules are shown right after login (cf. below).


Basic commands:

  • show available modules:
module avail

or

module avail <substring to be searched for>
  • show brief description for a given module:
module whatis <modulename>
  • (un)load a module:
module (un)load <modulename>
  • swap modules:
module swap <old module> <new module>
  • show loaded modules:
module list
  • show help for a specific module:
module help <modulename>
  • search for keywords (e.g. “compiler”, “debugger”, etc.) in module descriptions:
 module key <keyword>


Please have a look on here w.r.t. further functionality.


In order to work properly, applications built with a given compiler and MPI need to be linked with libraries built with the very same compiler and MPI. The module mechanism deployed on Hawk supports you in doing so by providing only those modules built with the currently loaded compiler and MPI. In case of swapping the compiler and/or MPI, available and loaded modules are also swapped in order to match with the new compiler/MPI.


Workflow: Right after login, you can load so-called system and core modules only:

hpcoft28@login2:~$ module avail

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32        intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                        intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver/9.7.0
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 


After loading a compiler, module avail shows also the modules depending on that compiler:

hpcoft28@login2:~$ module load gnu/9.1.0 
hpcoft28@login2:~$ module avail

-------- Compiler-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
   flex/2.6.4     metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20         superlu/4.3
   glm/0.9.9.5    metis/5.1.0-int32           metis/5.1.0-int64        (D)    openmpi/4.0.2    superlu/5.2.1 (D)

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                 (L)    intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 


In this set, you can - in particular - find MPI implementations. After loading one of those, module avail also shows the modules depending on the loaded compiler as well as MPI:

hpcoft28@login2:~$ module load mpt/2.20 
hpcoft28@login2:~$ module avail

-- MPI-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/mpt/2.20-44w73ly/gcc/9.1.0) --
   adios2/2.5.0                    mumps/5.2.1-int64           (D)    parmgridgen/1.0
   boost/1.70.0                    netcdf-cxx4/4.3.1                  petsc/3.12.2-int64-shared
   cgns/3.4.0-int32                netcdf-fortran/4.5.2               petsc/3.12.2-int64                                          (D)
   cgns/3.4.0-int64         (D)    netcdf/4.5.0                       scalapack/2.1.0-shared
   fftw/3.3.8                      netcdf/4.7.3                (D)    scalapack/2.1.0                                             (D)
   hdf5/1.10.5                     parallel-netcdf/1.11.2             scotch/6.0.9-int32-shared
   kahip/2.12-int32                parallel-netcdf/1.12.0      (D)    scotch/6.0.9-int32
   kahip/2.12-int64         (D)    parmetis/4.0.3-int32-shared        scotch/6.0.9-int64-shared
   mumps/5.2.1-int32-shared        parmetis/4.0.3-int32               scotch/6.0.9-int64                                          (D)
   mumps/5.2.1-int32               parmetis/4.0.3-int64-shared        trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0
   mumps/5.2.1-int64-shared        parmetis/4.0.3-int64        (D)    trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0 (D)

-------- Compiler-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
   flex/2.6.4     metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20      (L)    superlu/4.3
   glm/0.9.9.5    metis/5.1.0-int32           metis/5.1.0-int64        (D)    openmpi/4.0.2        superlu/5.2.1 (D)

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                 (L)    intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver/9.7.0
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 



module spider: In order to list all available modules without considering their dependencies, use module spider:

hpcoft28@login2:~$ module spider

------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
------------------------------------------------------------------------------------------------------------------------------------------------------
  adios2: adios2/2.5.0

  advisor: advisor/2019.4

  aocc: aocc/1.3.0, aocc/2.0.0, aocc/2.1.0

  boost: boost/1.70.0

  cgns: cgns/3.4.0-int32, cgns/3.4.0-int64

  cmake: cmake/3.15.2

  development/binutils: development/binutils/2.32

  fftw: fftw/3.3.8

  flex: flex/2.6.4

  forge: forge/19.1.2

  glm: glm/0.9.9.5

  gnu: gnu/9.1.0, gnu/9.2.0

  hdf5: hdf5/1.10.5

  inspector: inspector/2019.4

  intel: intel/18.0.2, intel/19.0.4, intel/19.1.0

  kahip: kahip/2.12-int32, kahip/2.12-int64

  metis: metis/5.1.0-int32-shared, metis/5.1.0-int32, metis/5.1.0-int64-shared, metis/5.1.0-int64

  mpt: mpt/2.20

  mumps: mumps/5.2.1-int32-shared, mumps/5.2.1-int32, mumps/5.2.1-int64-shared, mumps/5.2.1-int64

  netcdf: netcdf/4.5.0, netcdf/4.7.3

  netcdf-cxx4: netcdf-cxx4/4.3.1

  netcdf-fortran: netcdf-fortran/4.5.2

  openmpi: openmpi/4.0.2

  papi: papi/5.7.0

  parallel-netcdf: parallel-netcdf/1.11.2, parallel-netcdf/1.12.0

  parmetis: parmetis/4.0.3-int32-shared, parmetis/4.0.3-int32, parmetis/4.0.3-int64-shared, parmetis/4.0.3-int64

  parmgridgen: parmgridgen/1.0

  perfboost: perfboost

  perfcatcher: perfcatcher

  petsc: petsc/3.12.2-int64-shared, petsc/3.12.2-int64

  pgi: pgi/19.5, pgi/19.7

  python: python/3.8

  scalapack: scalapack/2.1.0-shared, scalapack/2.1.0

  scotch: scotch/6.0.9-int32-shared, scotch/6.0.9-int32, scotch/6.0.9-int64-shared, scotch/6.0.9-int64

  superlu: superlu/4.3, superlu/5.2.1

  system/site_names: system/site_names

  system/wrappers: system/wrappers/1.0

  system/ws: system/ws/f7ba528

  trilinos: trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0, trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0

  unsupported-modules: unsupported-modules

  uprof: uprof/3.1.35, uprof/3.2.228

  use.own: use.own

  vampir: vampir/9.7.0

  vampirserver: vampirserver/9.7.0

  vtune: vtune/2019.4

------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

------------------------------------------------------------------------------------------------------------------------------------------------------

 

hpcoft28@login2:~$ 


By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module:

hpcoft28@login2:~$ module spider fftw/3.3.8

------------------------------------------------------------------------------------------------------------------------------------------------------
  fftw: fftw/3.3.8
------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "fftw/3.3.8" module is available to load.

      gnu/9.1.0  mpt/2.20
      gnu/9.1.0  openmpi/4.0.2
 
    Help:
      FFTW is a C subroutine library for computing the discrete Fourier
      transform (DFT) in one or more dimensions, of arbitrary input size, and
      of both real and complex data (as well as of even/odd data, i.e. the
      discrete cosine/sine transforms or DCT/DST). We believe that FFTW, which
      is free software, should become the FFT library of choice for most
      applications.


 

hpcoft28@login2:~$ 



Environment changes: In case a user swaps the loaded compiler or MPI implementation, the module system tries to automatically reload matching modules from the compiler- and MPI-dependent sections. If there is no matching module, the respective module is set to the inactive state.
However, HLRS aspires to provide modules for as many compiler/MPI combinations as possible. From time to time, you should make use of this by profiling your code for different compiler/MPI combinations. Using different compilers and MPI implementations might significantly reduce the runtime of your code and is almost for free!


Details

  • It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.
  • If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). In fact, the rule is more complicated. Please refer to here with respect to details.
  • All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!
  • We provide support for Bash only. In case of the module command not being defined (correctly), cf. here and here.
  • Do an export LMOD_PAGER=None to turn off displaying multi-page outputs with a pager (e.g. less).




Hints for modulefile providers

  • Instead of the traditional one (Link, to be called Tmod hereinafter) Hawk uses a new implementation of the module command called Lmod (Link). While Tmod uses modulefiles written in TCL, Lmod uses modulefiles written in Lua.
  • Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. here and here)!
  • It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by
$(dirname ${LMOD_CMD})/tcl2lua.tcl <TCL-modulefile>
  • Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.
  • In case you install modules without Spack, please take care to place the respective modulefiles in the following locations:
    • modules without dependencies: /opt/hlrs/modulefiles_lmod/Core/
    • modules depending on compiler: /opt/hlrs/modulefiles_lmod/<compiler>/<compiler version>/
    • modules depending on compiler and MPI: /opt/hlrs/modulefiles_lmod/<MPI implementation>/<MPI version>-<hash>/<compiler>/<compiler version>
    • The above directories should already exist, so you don't have to care about the correct <hash>.


Details

  • Spider Cache : If there is a system spider cache, please recreate it in case you added / modified a modulefile! You can do this by calling $(dirname ${LMOD_CMD})/update_lmod_system_cache_files (cf. here).
  • PATH-like variables: It is important how duplicates in the PATH-like variables are handled. At the moment, LMOD_TMOD_PATH_RULE is set to yes in order to have the same behavior as the traditional module command (cf. here). Please have in mind that duplicates might be generated by:
    • Explicit loads of modules
    • Implicit loads due to modules loading further modules as prerequisites
    • Different modules inserting the same path
  • 2-digit-rule: Lmod developers suggest to assume compatibility in between compilers and MPI stacks which differ in the third digit of the version number only.
  • Shared home: Some care has to be taken if running Lmod on multiple clusters with shared home filesystems (as is e.g. w.r.t. Hawk and Vulcan), cf. here.

In case of problems, contact Björn Dick <dick@hlrs.de>.


Further nice features

  • Extensions: It’s possible to define “extensions”, i.e. some kind of pseudo-modules for stuff like numpy or scipy included in Python (but also e.g. FFTW included in AOCL, stuff included in Cray’s libsci, etc.). Those extensions will be listed by module avail and module spider in (almost) the same way as regular modules are listed. However, extensions can’t be loaded in the same way as regular modules, but Lmod tells the user how to do so after module load <extension-name> fails. With respect to details cf. here.
  • Nag message: It’s possible to print messages when loading modules, e.g. to announce deprecation (cf. here and here).
  • Brief description of module: If adding whatis("Description: foobar"), the string “foobar” will be shown in the output of module spider below the module name / versions (cf. here).
  • It’s possible to hide modulefiles and entire directories in the output of module avail and module spider by means of a leading dot in the file-/directory-name (cf. here).
  • Hierarchy labels: In case you add a compiler or MPI implementation, take care to add the respective path to $(dirname $LMOD_CMD)/../init/hlrs/SitePackage.lua in order to prettily label the group of modules build with that compiler / MPI (cf. here).
  • Debugging: Actions triggered by Lmod can be shown via module --trace <subcommand> (cf. here).




Hints for system administrators

  • Sticky modules: It’s possible to (almost) force particular packages to be always loaded (aka “sticky modules”, cf. here).
  • Auto-completion: Setting LMOD_EXACT_MATCH=yes disables automatic selection of default versions, but prevents from selecting modules by specifying a prefix only (cf. here).
  • Module categories: module keyword allows for searching strings in the whatis-text (cf. here). By adding category names in the whatis-text, this feature can hence be used to implement module categories as known from Hazel Hen.
  • Export module list: The spider tool can be used to export a list of available packages (e.g. for Wiki / Platform-Homepage, cf. here).
  • It’s possible to track module usage (cf. here).
  • It’s possible to use BLAS/LAPACK as an additional hierarchy layer (cf. here). We decided to use MKL as default for a start and evaluate performance of MKL vs. AOCL vs. PGI-version of BLAS/LAPACK vs. OpenBLAS later. Depending on the outcome, we might introduce an additional hierarchy layer in the future.