Extending the software stack on Hunter with Spack

This page is work in progress!!!

In its current form, this page is meant for HLRS staff. Expect thing to change and break at any time.

Introduction

Most of the software stack on Hunter is built on top of Spack. Once released, a software stack is immutable. Also, during development of a new software stack, working directly on the files and directories is strongly discouraged. The normal workflow is to use Spack's capabilities to "chain" a private Spack instance to an upstream instance (i.e. the HLRS release). This allows to do modifications, tests, and adjustments to your private Spack instance without interfering with the upstream instance. Once you are satisfied with your modifications, it is possible to incorporate them and thus extend the upstream instance without mutating existing state.

Pre-requisites

1. working access to internet from Hunter, e.g. ssh reverse tunnel with proxy setup

Spack Module

Spack is available after loading the module spack-user. The module however, can only be loaded if the environment variables TMPDIR and SPACK_USER_PREFIX are exported and point to existing directories. Spack will be configured to expect and place most of its data in the directory $SPACK_USER_PREFIX.

$ export TMPDIR=/localscratch/${UID}/spack-tmpdir # currently: export TMPDIR=${XDG_RUNTIME_DIR}/spack-tmpdir

$ HLRS_SOFTWARE_STACK_RELEASE_VERSION=hunter-CPE-24.07
$ export SPACK_USER_PREFIX=${HOME}/my-spack
$ mkdir -p $TMPDIR; mkdir -p $SPACK_USER_PREFIX
$ module use ${HLRS_OPT_PREFIX}/testing/release/${HLRS_SOFTWARE_STACK_RELEASE_VERSION}/modulefiles

$ module load spack-user

Essentially, the module does two things:

1. set the environment variable SPACK_ROOT pointing to the base of HLRS' Spack instance, and
2. provide the command alias spack which wraps the original Spack command and adds some additional parameters to pick up relevant configuration files.

$ alias spack
alias spack='spack -C $SPACK_ROOT/../config-hlrs -C $SPACK_ROOT/../config-user'

The first time you use Spack it is recommended to bootstrap it. This may take up to several minutes

$ spack bootstrap now

Your private Spack environment

All your private configuration and customization needs to be done in a so-called Spack environment. These are essentially directories below $SPACK_USER_PREFIX/environments. You may create multiple environments, but only one can be active at any given time. Creating an environment takes a somewhat lengthy command:

$ spack env create --include-concrete $SPACK_ROOT/../environments/hlrs --without-view my-tools

==> Created environment my-tools in: .../my-spack/environments/my-tools

==> Activate with: spack env activate my-tools

This will create the environment labeled my-tools in the indicated directory.

The option --include-concrete is what ties your environment to the HLRS software stack and must therefore not be omitted.

Next, you need to activate your environment. Unlike the message above, activation requires a slightly different command

$ eval $(spack env activate my-tools --sh)

Deactivate a Spack environment with the command despacktivate.

Note, that activating a Spack environment conveniently sets the shell environment (yes, too many "environments") variable SPACK_ENV, which points to the environments directory. In our case this is $SPACK_USER_PREFIX/environments/my-tools.

All configuration is done in the file spack.yaml. You may take inspiration from our template $SPACK_ROOT/../templates/user-env-spack.yaml.

$ cat $SPACK_ENV/spack.yaml

$ cat $SPACK_ROOT/../templates/user-env-spack.yaml
$ # cp $SPACK_ROOT/../templates/user-env-spack.yaml $SPACK_ENV/spack.yaml

$ vi $SPACK_ENV/spack.yaml

After editing your environments to add specs, it is time to concretize your environment. Concretization will inspect all software packages installed already in the HLRS software stack and determine what is messing and which version to install.

spack concretize --reuse > $SPACK_ENV/concretize.log
less $SPACK_ENV/concretize.log

We recommend to keep a log of the concretization step as shown above. Make sure to check that the dependency tree agrees with your expectations.

Finally, it is time to build and install

spack install