Extending the software stack on Hunter with Spack

This page is work in progress!!!

In its current form, this page is meant for HLRS staff. Expect thing to change and break at any time.

Introduction

Most of the software stack on Hunter is built on top of Spack. Once released, a software stack is immutable. Also, during development of a new software stack, working directly on the files and directories is strongly discouraged. The normal workflow is to use Spack's capabilities to "chain" a private Spack instance to an upstream instance (i.e. the HLRS release). This allows to do modifications, tests, and adjustments to your private Spack instance without interfering with the upstream instance. Once you are satisfied with your modifications, it is possible to incorporate them and thus extend the upstream instance without mutating existing state.

Pre-requisites

1. working access to internet from Hunter, e.g. ssh reverse tunnel with proxy setup

Spack Module

Spack is available after loading the module spack-user. The module however, can only be loaded if the environment variables TMPDIR and SPACK_USER_PREFIX are exported and point to existing directories. Spack will be configured to expect and place most of its data in the directory $SPACK_USER_PREFIX.

module load HLRS
export TMPDIR=/localscratch/${UID}/spack-tmpdir
export SPACK_USER_PREFIX=${HOME}/my-spack
mkdir -p $TMPDIR; mkdir -p $SPACK_USER_PREFIX
module use ${HLRS_SOFTWARE_STACK_RELEASE_ROOT}/modulefiles
module load spack-user
cd $SPACK_USER_PREFIX

Essentially, the module does two things:

1. set the environment variable SPACK_ROOT pointing to the base of HLRS' Spack instance, and
2. provide the command alias spack which wraps the original Spack command and adds some additional parameters to pick up relevant configuration files.

alias spack
# Output should show: alias spack='spack -C $SPACK_ROOT/../config-hlrs -C $SPACK_ROOT/../config-user'

The first time you use Spack it is recommended to bootstrap it. This may take up to several minutes

spack bootstrap now

Your private Spack environment

All your private configuration and customization needs to be done in a so-called Spack environment. These are essentially directories below $SPACK_USER_PREFIX/environments. You may create multiple environments, but only one can be active at any given time. We have provided a template for environment configuration. Create your environments by copying the template into the appropriate directory:

mkdir $SPACK_USER_PREFIX/environments; cd $SPACK_USER_PREFIX/environments
mkdir my-tools           # for an environment named "my-tools"
cp $SPACK_ROOT/../templates/user-env-spack.yaml my-tools/spack.yaml
cat my-tools/spack.yaml
# Output should include:
# ...
# include_concrete:
#   - $spack/../environments/hlrs

The option setting include-concrete: is what ties your environment to the HLRS software stack and must therefore not be removed.

Note, that creating environments with the Spack command spack env create --include-concrete $SPACK_ROOT/... does not work because of file permission issues.

Next, you need to activate your environment.

eval $(spack env activate my-tools --sh)

Deactivate a Spack environment with the command despacktivate.

Note, that activating a Spack environment conveniently sets the shell environment (yes, too many "environments") variable SPACK_ENV, which points to the directory of the currently active environment. In our case this is $SPACK_USER_PREFIX/environments/my-tools.

All configuration is done in the file spack.yaml. You may take inspiration from our template $SPACK_ROOT/../templates/user-env-spack.yaml.

cat $SPACK_ENV/spack.yaml
cat $SPACK_ROOT/../templates/user-env-spack.yaml
vi $SPACK_ENV/spack.yaml

After editing your environments to add specs, it is time to concretize your environment. Concretization will inspect all software packages installed already in the HLRS software stack and determine what is messing and which version to install.

spack concretize > $SPACK_ENV/concretize.log
less $SPACK_ENV/concretize.log

We recommend to keep a log of the concretization step as shown above. Make sure to check that the dependency tree agrees with your expectations.

Finally, it is time to build and install

spack install