- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Module command: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
(Use unique variable name to fix issue if this workaround is used in multiple .modulerc files.)
m (only bash supported)
 
(4 intermediate revisions by 2 users not shown)
Line 3: Line 3:
Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as <tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, <tt>MANPATH</tt>, etc. Modulefiles may be shared by many users on a system and users may have their own collection to supplement or replace the shared modulefiles.
Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as <tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, <tt>MANPATH</tt>, etc. Modulefiles may be shared by many users on a system and users may have their own collection to supplement or replace the shared modulefiles.


Modules can be loaded and unloaded dynamically and atomically, in a clean fashion. All popular shells are supported, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl.
Modules can be loaded and unloaded dynamically and atomically, in a clean fashion.
Even though all popular shells, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl should work, only bash will be supported by HLRS.


Modules are useful in managing different versions of applications. Modules can also be bundled into metamodules that will load an entire suite of different applications.
Modules are useful in managing different versions of applications. Modules can also be bundled into metamodules that will load an entire suite of different applications.
Line 17: Line 18:
  <pre>module help modulecommand</pre> The ''help'' command will provide more detailed information on the specified module. Without argument ''modulecommand'' you will get online help for the ''module'' command.
  <pre>module help modulecommand</pre> The ''help'' command will provide more detailed information on the specified module. Without argument ''modulecommand'' you will get online help for the ''module'' command.


  <pre>module avail</pre> The ''avail'' option displays all the modules that are available on the system. Where there is more than one version of a module, the default version is denoted by (default).
  <pre>module avail</pre> The ''avail'' option displays all the modules that are available on the system. Where there is more than one version of a module, the default version is denoted by (default).
 
<pre>module avail prod / module avail -S prod </pre> List all the modules starting with ''prod''. When added the '''-S''' option also modules are listed containing ''prod''.


  <pre>module list</pre> The ''list'' option displays all the modules that are currently loaded into your user environment.
  <pre>module list</pre> The ''list'' option displays all the modules that are currently loaded into your user environment.
Line 27: Line 30:
  <pre>module display modulename</pre> The ''display'' option shows the changes that the specified module will make in your environment, for example, what will be added to the PATH and MANPATH environment variables.
  <pre>module display modulename</pre> The ''display'' option shows the changes that the specified module will make in your environment, for example, what will be added to the PATH and MANPATH environment variables.


  <pre>module switch modulename/currentversionmodulename/newversion</pre> The ''switch'' option replaces the currently loaded version of a module with a different version. When the new verion is loaded, the man page for the specified software will also be updated.  
  <pre>module switch modulename/currentversion  modulename/newversion</pre> The ''switch'' option replaces the currently loaded version of a module with a different version. When the new verion is loaded, the man page for the specified software will also be updated.  


</ul>
</ul>
Line 48: Line 51:
</ul>
</ul>


== build your own modules ==
=== automated way ===
You can easily create modulefiles of an application or library yourself. CRAY provides the tool '''craypkg-gen''' to create modulesfiles as well as corresponding .pc files for libraries (as well as rpm packages).
Assuming your application/library is located in directory ''<PathToApp>/PRODUCT/VERS/'' you crate these files by:


== Examples ==
<pre>$> module load craypkg-gen
$> craypkg-gen -p <PathToApp>/PRODUCT/VERS/  # to create the pc-files
$> craypkg-gen -m <PathToApp>/PRODUCT/VERS/  # to create the modulefile
</pre>
this creates .pc files in the lib directories of your application/library and a directory structure in ''<PathToApp>/modulefiles''. This directory and sub-directory, should contain only modulefiles. You can add these modules to the module environment by using
<pre>$> module use <PathToApp>/modulefiles</pre>
And load one of this modules by:
<pre>$> module load PRODUCT/VERS</pre>


When writing module files normal Tcl commands can be used to do things like gnerating lock- or configuration-files. See the Examples below.
=== manual creation / editing ===
Module files contain TCL commands to control the user environments, like setting environment variables and paths, see the Examples below.


<b>Generating a default configuration file</b>
==== modulefile Example ====
<pre>
<pre>
#%Module
#%Module
Line 91: Line 106:
}
}
</pre>
</pre>
==== Example: mytool library ====
Assuming a library libmyXY.a is located in <PathToApp>/my_tool/v.1/lib/. First, the .pc file (myXY.pc) has to be created in <PathToApp>/my_tool/v.1/lib/pkgconfig, containing
<pre>
prefix=/path/to/my_tool/v.1
exec_prefix=${prefix}
includedir=${prefix}/include
libdir=${exec_prefix}/lib
Name: myXY
Description: The myXY library
Version: 1.0.0
Cflags: -I${includedir}/myXY.c
Libs: -L${libdir} -lmyXY
</pre>
This can be done for all the libraries (e.g. also myYZ and myZZ) provided by this application/module.
Furthermore, a module file has to be created/modified containing:
<pre>
#%Module
...
append-path my_tool_PKGCONFIG_LIBS {myXY:myYZ:myZZ}
append-path PKG_CONFIG_PRODUCTS {my_tool}
append-path PKG_CONFIG_PATH <PathToApp>/my_tool/v.1/lib/pkgconfig
append-path LD_LIBRARY_PATH <PathToApp>/my_tool/v.1/lib
# assuming there are also binaries which should get available
append-path PATH <PathToApp>/my_tool/v.1/bin
# assuming there are manpages available
append-path MANPATH <PathToApp>/my_tool/v.1/man
</pre>
As a result, the compiler/include/linker flags (e.g. ''-lmyXY'') and the corresponding include/library paths are automatically added while compiling and linking.


== Global default version ==
== Global default version ==

Latest revision as of 13:24, 19 May 2023

The Environment Modules package provides for the dynamic modification of a user's environment via modulefiles.

Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as PATH, LD_LIBRARY_PATH, MANPATH, etc. Modulefiles may be shared by many users on a system and users may have their own collection to supplement or replace the shared modulefiles.

Modules can be loaded and unloaded dynamically and atomically, in a clean fashion. Even though all popular shells, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl should work, only bash will be supported by HLRS.

Modules are useful in managing different versions of applications. Modules can also be bundled into metamodules that will load an entire suite of different applications.

Usage

The environmental setting using the Environment Modules Project method will not be saved and will be lost for a new session. A new session (login, new job) will have the default environment. The Cluster system uses modules in the user environment to support multiple versions of software, such as compilers, and to create integrated software packages. As new versions of the supported software become available, they are added automatically to the programming environment, while earlier versions are retained to support legacy applications. By specifying the module to load, you can choose the default version of an application, or another version. Modules also porvide a simple mechanism for updating certain environment variables, such as PATH, MANPATH, and LD_LIBRARY_PATH.
The following topics describe the fundamentals of using the modules environment.

  • to invoke the module command, type:
    module option args
    module help modulecommand
    The help command will provide more detailed information on the specified module. Without argument modulecommand you will get online help for the module command.
    module avail
    The avail option displays all the modules that are available on the system. Where there is more than one version of a module, the default version is denoted by (default).
    module avail prod / module avail -S prod 
    List all the modules starting with prod. When added the -S option also modules are listed containing prod.
    module list
    The list option displays all the modules that are currently loaded into your user environment.
    module add / module load modulename
    The add option and the load option have the same function - to load the specified module into your user environment.
    module rm / module unload modulename
    The rm option and the unload option have the same function - to unload the specified module from your user environment. Before loading a module that replaces another version of the same package, you should always unload the module that is to be replaced.
    module display modulename
    The display option shows the changes that the specified module will make in your environment, for example, what will be added to the PATH and MANPATH environment variables.
    module  switch  modulename/currentversion  modulename/newversion
    The switch option replaces the currently loaded version of a module with a different version. When the new verion is loaded, the man page for the specified software will also be updated.
  • using $HOME/.modulerc
    This file can be used to load or to define your own environment during each login. An example looks like this:
    #%Module1.0#
    
    set version 1.0
    module load use.own
    

    The module use.own will add $HOME/privatemodules to the list of directories that the module command will search for modules. Place your own module files here. This module, when loaded, will create this directory if necessary.

    see also:

     man module
    

build your own modules

automated way

You can easily create modulefiles of an application or library yourself. CRAY provides the tool craypkg-gen to create modulesfiles as well as corresponding .pc files for libraries (as well as rpm packages). Assuming your application/library is located in directory <PathToApp>/PRODUCT/VERS/ you crate these files by:

$> module load craypkg-gen
$> craypkg-gen -p <PathToApp>/PRODUCT/VERS/   # to create the pc-files
$> craypkg-gen -m <PathToApp>/PRODUCT/VERS/   # to create the modulefile 

this creates .pc files in the lib directories of your application/library and a directory structure in <PathToApp>/modulefiles. This directory and sub-directory, should contain only modulefiles. You can add these modules to the module environment by using

$> module use <PathToApp>/modulefiles

And load one of this modules by:

$> module load PRODUCT/VERS

manual creation / editing

Module files contain TCL commands to control the user environments, like setting environment variables and paths, see the Examples below.

modulefile Example

#%Module
#
# --------------------------------------------------------
# -- Set path to user's home -----------------------------
set  DIR_NAME  ~
#
# -- Test whether the file is a regular file -------------
set  PROPS_EXIST  [file isfile $DIR_NAME/.dcr/dicom.props]
#
# -- If the file is already there PROPS_EXIST will be 1 --
if {$PROPS_EXIST != 1} {
   set dir_d  [file mkdir $DIR_NAME/.dcr]
   set file_d [open $DIR_NAME/.dcr/dicom.props w 0444]
   puts $file_d "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
   puts $file_d "  <jobProperties>"
   puts $file_d "  <finishedJobs length=\"0\" />"
   ...
   puts $file_d "</jobProperties>"
}
#
set           DCR_BASE      /opt/bwgrid/stuttgart/dicom-toolbox
#
append-path   DCR_JARPATH   $DCR_BASE/reconstruction/jar
append-path   PATH          $DCR_BASE
#
# -------------------------------------------------------
#
module-whatis "HLRS -- Dicom-Toolbox"
#
# -------------------------------------------------------
#
proc ModulesHelp { } {
        global version
        puts stderr "\tdicom-toolbox - invokes the HLRS Dicom-Toolbox"
}

Example: mytool library

Assuming a library libmyXY.a is located in <PathToApp>/my_tool/v.1/lib/. First, the .pc file (myXY.pc) has to be created in <PathToApp>/my_tool/v.1/lib/pkgconfig, containing

 
 prefix=/path/to/my_tool/v.1
 exec_prefix=${prefix} 
 includedir=${prefix}/include 
 libdir=${exec_prefix}/lib 

 Name: myXY
 Description: The myXY library 
 Version: 1.0.0 
 Cflags: -I${includedir}/myXY.c 
 Libs: -L${libdir} -lmyXY

This can be done for all the libraries (e.g. also myYZ and myZZ) provided by this application/module. Furthermore, a module file has to be created/modified containing:

 
#%Module
...
 append-path my_tool_PKGCONFIG_LIBS {myXY:myYZ:myZZ}
 append-path PKG_CONFIG_PRODUCTS {my_tool}
 append-path PKG_CONFIG_PATH <PathToApp>/my_tool/v.1/lib/pkgconfig
 append-path LD_LIBRARY_PATH <PathToApp>/my_tool/v.1/lib
 # assuming there are also binaries which should get available
 append-path PATH <PathToApp>/my_tool/v.1/bin
 # assuming there are manpages available
 append-path MANPATH <PathToApp>/my_tool/v.1/man

As a result, the compiler/include/linker flags (e.g. -lmyXY) and the corresponding include/library paths are automatically added while compiling and linking.

Global default version

A default version for a specific module can be specified using the module-version command in .modulerc files. In general the module-version command can be used to define additional short hand symbols for any module. To set e.g. the default version for the module mpi/openmpi to 1.6.4-gnu-4.7.2 and a testing version mpi/openmpi/1.7.1-gnu-4.7.2 create the file $MODULEPATH/mpi/openmpi/.modulerc with the following content:

File: .modulerc
#%Module1.0
set DEFAULT  1.6.4-gnu-4.7.2
module-version openmpi openmpi/$DEFAULT

# circumvent problem with duplicate definition of symbol testing
# The use variable name has to be unique to prevent conflicts if
# this workaround is used in multiple .modulerc files.
if { ![info exists MPI_OPENMPI_TESTING] } {
  set MPI_OPENMPI_TESTING   1.7.1-gnu-4.7.2
  module-version mpi/openmpi/$MPI_OPENMPI_TESTING    testing
}


Bash completion

Since modules version 3.2.7 a bash completion module can be found under $MODULESHOME/init.

Copy this file into $HOME/.bash_completion.d/

External links