- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Module environment(Hawk): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
Line 51: Line 51:
   advisor/2019.4        development/binutils/2.32        intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   advisor/2019.4        development/binutils/2.32        intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                        intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0-hmpt-2.19-gnu-4.8.5
   aocc/2.0.0            gnu/9.1.0                        intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
Line 82: Line 82:
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0-hmpt-2.19-gnu-4.8.5
   aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
Line 126: Line 126:
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher        uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/1.3.0            forge/19.1.2                    intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0-hmpt-2.19-gnu-4.8.5
   aocc/2.0.0            gnu/9.1.0                (L)    intel/19.1.0 (D)    pgi/19.7    (D)    vampirserver/9.7.0
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   aocc/2.1.0    (D)    gnu/9.2.0                (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
   cmake/3.15.2          inspector/2019.4                perfboost          uprof/3.1.35
Line 238: Line 238:
   vampir: vampir/9.7.0
   vampir: vampir/9.7.0


   vampirserver: vampirserver/9.7.0-hmpt-2.19-gnu-4.8.5
   vampirserver: vampirserver/9.7.0


   vtune: vtune/2019.4
   vtune: vtune/2019.4

Revision as of 16:46, 15 January 2020

Hints for users

A module environment is used on Hawk in order to allow for flexibly choosing software to be used out of a large set. In order to use a particular software, you first have to load the respective module. A set of default modules is already loaded after login.


A new implementation of the module command is used on Hawk in order to allow for improved user experience. However, the same sub-commands are available as known from former systems. The layout of outputs yet slightly differs. Furthermore, not all available modules are shown right after login (cf. below).


Basic commands:

  • show available modules:
module avail

or

module avail <substring to be searched for>
  • show brief description for a given module:
module whatis <modulename>
  • (un)load a module:
module (un)load <modulename>
  • swap modules:
module swap <old module> <new module>
  • show loaded modules:
module list
  • show help for a specific module:
module help <modulename>
  • search for keywords (e.g. “compiler”, “debugger”, etc.) in module descriptions:
 module key <keyword>


Please have a look on here w.r.t. further functionality.


In order to work properly, applications built with a given compiler and MPI need to be linked with libraries built with the very same compiler and MPI. The module mechanism deployed on Hawk supports you in doing so by providing only those modules built with the currently loaded compiler and MPI. In case of swapping the compiler and/or MPI, available and loaded modules are also swapped in order to match with the new compiler/MPI.


Right after login, you can load so-called system and core modules only:

hpcoft28@login2:~$ module avail

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32        intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                        intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver/9.7.0
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 


After loading a compiler, module avail shows also the modules depending on that compiler:

hpcoft28@login2:~$ module load gnu/9.1.0 
hpcoft28@login2:~$ module avail

-------- Compiler-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
   flex/2.6.4     metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20         superlu/4.3
   glm/0.9.9.5    metis/5.1.0-int32           metis/5.1.0-int64        (D)    openmpi/4.0.2    superlu/5.2.1 (D)

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                 (L)    intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 


In this set, you can - in particular - find MPI implementations. After loading one of those, module avail also shows the modules depending on the loaded compiler as well as MPI:

hpcoft28@login2:~$ module load mpt/2.20 
hpcoft28@login2:~$ module avail

-- MPI-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/mpt/2.20-44w73ly/gcc/9.1.0) --
   adios2/2.5.0                    mumps/5.2.1-int64           (D)    parmgridgen/1.0
   boost/1.70.0                    netcdf-cxx4/4.3.1                  petsc/3.12.2-int64-shared
   cgns/3.4.0-int32                netcdf-fortran/4.5.2               petsc/3.12.2-int64                                          (D)
   cgns/3.4.0-int64         (D)    netcdf/4.5.0                       scalapack/2.1.0-shared
   fftw/3.3.8                      netcdf/4.7.3                (D)    scalapack/2.1.0                                             (D)
   hdf5/1.10.5                     parallel-netcdf/1.11.2             scotch/6.0.9-int32-shared
   kahip/2.12-int32                parallel-netcdf/1.12.0      (D)    scotch/6.0.9-int32
   kahip/2.12-int64         (D)    parmetis/4.0.3-int32-shared        scotch/6.0.9-int64-shared
   mumps/5.2.1-int32-shared        parmetis/4.0.3-int32               scotch/6.0.9-int64                                          (D)
   mumps/5.2.1-int32               parmetis/4.0.3-int64-shared        trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0
   mumps/5.2.1-int64-shared        parmetis/4.0.3-int64        (D)    trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0 (D)

-------- Compiler-dependent modules   (/lustre/cray/ws9/6/ws/hpcoft28-spack/opt-philipp/modulefile/lmod/linux-rhel7-x86_64/gcc/9.1.0) --------
   flex/2.6.4     metis/5.1.0-int32-shared    metis/5.1.0-int64-shared        mpt/2.20      (L)    superlu/4.3
   glm/0.9.9.5    metis/5.1.0-int32           metis/5.1.0-int64        (D)    openmpi/4.0.2        superlu/5.2.1 (D)

---------------------------------------------- Core Modules   (/opt/hlrs/modulefiles_lmod/Core) ----------------------------------------------
   advisor/2019.4        development/binutils/2.32 (L)    intel/18.0.2        perfcatcher         uprof/3.2.228                          (D)
   aocc/1.3.0            forge/19.1.2                     intel/19.0.4        pgi/19.5            vampir/9.7.0
   aocc/2.0.0            gnu/9.1.0                 (L)    intel/19.1.0 (D)    pgi/19.7     (D)    vampirserver/9.7.0
   aocc/2.1.0     (D)    gnu/9.2.0                 (D)    papi/5.7.0          python/3.8          vtune/2019.4
   cmake/3.15.2          inspector/2019.4                 perfboost           uprof/3.1.35

---------------------------------------------- System Modules   (/opt/hlrs/system/modulefiles) -----------------------------------------------
   system/site_names (L)    system/wrappers/1.0 (L)    system/ws/f7ba528 (L)    unsupported-modules    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


hpcoft28@login2:~$ 


In order to list all available modules without considering their dependencies, use module spider:

hpcoft28@login2:~$ module spider

------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
------------------------------------------------------------------------------------------------------------------------------------------------------
  adios2: adios2/2.5.0

  advisor: advisor/2019.4

  aocc: aocc/1.3.0, aocc/2.0.0, aocc/2.1.0

  boost: boost/1.70.0

  cgns: cgns/3.4.0-int32, cgns/3.4.0-int64

  cmake: cmake/3.15.2

  development/binutils: development/binutils/2.32

  fftw: fftw/3.3.8

  flex: flex/2.6.4

  forge: forge/19.1.2

  glm: glm/0.9.9.5

  gnu: gnu/9.1.0, gnu/9.2.0

  hdf5: hdf5/1.10.5

  inspector: inspector/2019.4

  intel: intel/18.0.2, intel/19.0.4, intel/19.1.0

  kahip: kahip/2.12-int32, kahip/2.12-int64

  metis: metis/5.1.0-int32-shared, metis/5.1.0-int32, metis/5.1.0-int64-shared, metis/5.1.0-int64

  mpt: mpt/2.20

  mumps: mumps/5.2.1-int32-shared, mumps/5.2.1-int32, mumps/5.2.1-int64-shared, mumps/5.2.1-int64

  netcdf: netcdf/4.5.0, netcdf/4.7.3

  netcdf-cxx4: netcdf-cxx4/4.3.1

  netcdf-fortran: netcdf-fortran/4.5.2

  openmpi: openmpi/4.0.2

  papi: papi/5.7.0

  parallel-netcdf: parallel-netcdf/1.11.2, parallel-netcdf/1.12.0

  parmetis: parmetis/4.0.3-int32-shared, parmetis/4.0.3-int32, parmetis/4.0.3-int64-shared, parmetis/4.0.3-int64

  parmgridgen: parmgridgen/1.0

  perfboost: perfboost

  perfcatcher: perfcatcher

  petsc: petsc/3.12.2-int64-shared, petsc/3.12.2-int64

  pgi: pgi/19.5, pgi/19.7

  python: python/3.8

  scalapack: scalapack/2.1.0-shared, scalapack/2.1.0

  scotch: scotch/6.0.9-int32-shared, scotch/6.0.9-int32, scotch/6.0.9-int64-shared, scotch/6.0.9-int64

  superlu: superlu/4.3, superlu/5.2.1

  system/site_names: system/site_names

  system/wrappers: system/wrappers/1.0

  system/ws: system/ws/f7ba528

  trilinos: trilinos/12.14.1-parallel-netcdf-1.11.2-netcdf-4.5.0, trilinos/12.14.1-shared-parallel-netcdf-1.11.2-netcdf-4.5.0

  unsupported-modules: unsupported-modules

  uprof: uprof/3.1.35, uprof/3.2.228

  use.own: use.own

  vampir: vampir/9.7.0

  vampirserver: vampirserver/9.7.0

  vtune: vtune/2019.4

------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

------------------------------------------------------------------------------------------------------------------------------------------------------

 

hpcoft28@login2:~$ 


By passing a specific module to the module spider command, the system tells you which modules have to be loaded first in order to load the respective module:

hpcoft28@login2:~$ module spider fftw/3.3.8

------------------------------------------------------------------------------------------------------------------------------------------------------
  fftw: fftw/3.3.8
------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "fftw/3.3.8" module is available to load.

      gnu/9.1.0  mpt/2.20
      gnu/9.1.0  openmpi/4.0.2
 
    Help:
      FFTW is a C subroutine library for computing the discrete Fourier
      transform (DFT) in one or more dimensions, of arbitrary input size, and
      of both real and complex data (as well as of even/odd data, i.e. the
      discrete cosine/sine transforms or DCT/DST). We believe that FFTW, which
      is free software, should become the FFT library of choice for most
      applications.


 

hpcoft28@login2:~$ 


Details

  • It’s not allowed to have more than one version of a module loaded at a time. In particular, it’s not possible to have more than one compiler loaded at once. In case you need multiple compilers, please inspect the respective module files and set the required environment variables by your own.
  • If no version is specified, the module command picks the one with the highest version. If there are multiple modules with the same version, it picks the one found first while searching “from top to bottom” (w.r.t. to the output of module avail). In fact, the rule is more complicated. Please refer to here with respect to details.
  • All Bash scripts should start with #!/bin/bash, not #!/bin/sh, otherwise the module command will not be available in the script!
  • We provide support for Bash only. In case of the module command not being defined (correctly), cf. here and here.
  • Do an export LMOD_PAGER=None to turn off displaying multi-page outputs with a pager (e.g. less).




Hints for modulefile providers

  • Instead of the traditional one (Link, to be called Tmod hereinafter) Hawk uses a new implementation of the module command called Lmod (Link). While Tmod uses modulefiles written in TCL, Lmod uses modulefiles written in Lua.
  • Lmod can however also read and use TCL modulefiles. It does this by evaluating the TCL statements as far as possible, fill the results into equivalent Lua statements and execute the resulting Lua script. This might lead to problems in special cases (cf. here and here)!
  • It’s also possible to convert traditional TCL modulefiles to their Lua equivalents by calling $(dirname ${LMOD_CMD})/tcl2lua.tcl.
  • Spack is configured to automatically generate modulefiles in TCL as well as Lua format. You hence don’t have to care about modulefiles when installing via Spack.


Details

  • Spider cache: If there is a system spider cache, please recreate it in case you added / modified a modulefile! You can do this by calling $(dirname ${LMOD_CMD})/update_lmod_system_cache_files (cf. here).
  • PATH-like variables: It is important how duplicates in the PATH-like variables are handled. At the moment, LMOD_TMOD_PATH_RULE is set to yes in order to have the same behavior as the traditional module command (cf. here). Please have in mind that duplicates might be generated by:
    • Explicit loads of modules
    • Implicit loads due to modules loading further modules as prerequisites
    • Different modules inserting the same path
  • 2-digit-rule: Lmod developers suggest to assume compatibility in between compilers and MPI stacks which differ in the third digit of the version number only.
  • Shared home: Some care has to be taken if running Lmod on multiple clusters with shared home filesystems (as is e.g. w.r.t. Hawk and Vulcan), cf. here.
  • In case of problems, contact Björn Dick <dick@hlrs.de>.


Further nice features

  • Extensions: It’s possible to define “extensions”, i.e. some kind of pseudo-modules for stuff like numpy or scipy included in Python (but also e.g. FFTW included in AOCL, stuff included in Cray’s libsci, etc.). Those extensions will be listed by module avail and module spider in (almost) the same way as regular modules are listed. However, extensions can’t be loaded in the same way as regular modules, but Lmod tells the user how to do so after module load <extension-name> fails. With respect to details cf. here.
  • Nag message: It’s possible to print messages when loading modules, e.g. to announce deprecation (cf. here and here).
  • Brief description of module: If adding whatis("Description: foobar"), the string “foobar” will be shown in the output of module spider below the module name / versions (cf. here).
  • It’s possible to hide modulefiles and entire directories in the output of module avail and module spider by means of a leading dot in the file-/directory-name (cf. here).
  • Hierarchy labels: In case you add a compiler or MPI implementation, take care to add the respective path to $(dirname $LMOD_CMD)/../init/hlrs/SitePackage.lua in order to prettily label the group of modules build with that compiler / MPI (cf. here).
  • Debugging: Actions triggered by Lmod can be shown via module --trace <subcommand> (cf. here).




Hints for system administrators

  • Sticky modules: It’s possible to (almost) force particular packages to be always loaded (aka “sticky modules”, cf. here).
  • Auto-completion: Setting LMOD_EXACT_MATCH=yes disables automatic selection of default versions, but prevents from selecting modules by specifying a prefix only (cf. here).
  • Module categories: module keyword allows for searching strings in the whatis-text (cf. here). By adding category names in the whatis-text, this feature can hence be used to implement module categories as known from Hazel Hen.
  • Export module list: The spider tool can be used to export a list of available packages (e.g. for Wiki / Platform-Homepage, cf. here).
  • It’s possible to track module usage (cf. here).
  • It’s possible to use BLAS/LAPACK as an additional hierarchy layer (cf. here). We decided to use MKL as default for a start and evaluate performance of MKL vs. AOCL vs. PGI-version of BLAS/LAPACK vs. OpenBLAS later. Depending on the outcome, we might introduce an additional hierarchy layer in the future.