Big Data, AI Aplications and Frameworks

Note: This page is being actively edited. Please don’t use links to its sections yet, as the content structure may change

Hardware overview

AI and Big Data (HPDA) workflows often require local storage, while HPC nodes usually do not have any local drive. Local storage is available on the nodes mentioned below. Make sure that your application uses correct path for local files (cache, scratch). /tmp (unless mounted as local SSD) is usually a very small in-memory filesystem.

Vulcan

Following nodes are to be used for AI and HPDA jobs:

• clx-21 - tuned for HPDA, no GPUs
• clx-ai - tuned for AI, 8 × V100 GPUs per node

Cray Urika-CS container can be executed on clx-ai and clx-21 nodes. For more info read corresponding wiki page.

Hawk

Following nodes can be used for AI jobs:

• hawk-ai - tuned for AI, 8 × A100 GPUs per node

Python and Python packages

There are three most popular way to manage Python packages for the projects you work on:

1. Using Conda
2. Using Virtualenv
3. Installing globally (for user) with pip install --user

Last one is less reproducible and may cause you troubles when working one several (sub-)projects.

Conda modules

Miniconda is available as Module, and can be used with packages from main and r channels.

Miniconda itself is distributed under 3-clause BSD License, but it allows users to install third-party software with proprietary licenses. You will have to explicitly accept this license when using Miniconda first time. Please read carefully the license and third-party licenses which are mentioned there (including Nvidia cuDNN license).

First run is slightly different on Vulcan and Hawk. (TLDR; call module load bigdata/conda/miniconda-4.10.3, and follow instructions)

Vulcan

module load bigdata/conda/miniconda-4.10.3

When loading the module a pager program will be started to display you license terms, after reading exit the pager (by default by pressing q) and enter yes or no to accept or decline the license. After accepting the license module will be loaded. Next time the module is loaded without further actions.

Hawk

Similar as for Vulcan, but with separate module and command at first usage:

module load bigdata/conda/miniconda-4.10.3-license
conda_license

After accepting the license use module load bigdata/conda/miniconda-4.10.3 to load the module.

Conda environments

After module is loaded you need to initialize conda with source activate [env-name]. If you ommit the env-name conda will activate the default (read only) base environment with a minimal set of packages.

Use Conda as usual. Only main and r channels are available.

In the environment files you will need to delete channels: section.

Conda will creatre environments in ~/.conda/envs and store packages cache in ~/.conda/pkgs. This folder can become quite big and exhaust your quota. Environment variables CONDA_ENVS_PATH and CONDA_PKGS_DIRS can help with this.

Here is some random example:

module load bigdata/conda
source activate
conda env list
conda create -n my-jupyter jupyter tensorflow-gpu pandas
rm -r ~/.conda/pkgs # delete cache
conda activate my-jupyter

Please note: Conda packages (e.g. Tensorflow) are compiled for a generic CPU.

virtualenv and pip

Packages installed with pip are often compiled during installation, pip can be used both with Conda and with virtualenv.

Here is an example how to create a virtual environment (not using conda).

module load python/3.8 # Load required python module (you can also use the system one, but this is less reproducible)
mkdir ~/venvs # directory for your environments
python3 -m venv ~/venvs/myproject # create the environment
source ~/venvs/myproject/bin/activate # activate environment to use it
which python3 # verify that you are using your environment

pip offline

…

Spark

Vulcan

Cray Urika-CS

Spark is deployed when you run Urika-CS in interactive mode (with start_analytics).

Bare-metal setup

Available on all compute nodes at Vulcan. This is a test-installation, which is not tuned yet.

To deploy Spark, run in your job script (or interactively):

module load bigdata/spark_cluster
init-spark

This will deploy Spark master on the current node and Spark workers on the rest of the nodes. Spark is started in the background, but you will see its output in the console.

init-spark script also creates $HOME/bigdata/$PBS_JOBID/ directory with configs and logs.

/tmp/${USER}_spark is used as SPARK_WORKER_DIR (local scratch for spark). Be aware: on most nodes /tmp is a ram-disk, and is quite small.

On clx-21 nodes you must set SPARK_WORKER_DIR before running init-spark:

module load bigdata/spark_cluster
export SPARK_WORKER_DIR="/localscratch/${PBS_JOBID}/spark_worker_dir"
mkdir -p "$SPARK_WORKER_DIR"
init-spark

Containers

Singularity

Singularity has been created as HPC aware containers platform. For more security we run Singularity in a rootless mode, in this mode SIF images are extracted into a sandbox directory, this requires nodes with local storage. Make sure to setup SINGULARITY_TMPDIR and SINGULARITY_CACHEDIR environment variables and create corresponding directories on a local drive.

Vulcan

To use Singularity containers (e.g. for Cray Urika-CS) add UNS=true to qsub selectors. Currently only a preconfigured for Cray Urika-CS version of Singularity is preinstalled. Nodes with Singularity support are clx-21 and clx-ai. Local NVME drives are mounted as /localscratch on these nodes.

Please create a working directory mkdir -p "/localscratch/${PBS_JOBID}" to make it consistent with Hawk (see below).

Be aware local storage is wiped after your job ends!

Hawk

Singularity containers on Hawk can only be executed on the AI nodes. To use Singularity containers add UNS=true to qsub selectors.

Nodes with Singularity support are rome-ai. Local NVME drives are mounted as /localscratch on these nodes. Users have write permissions under /localscratch/${PBS_JOBID}/

Be aware local storage is wiped after your job ends!

Singularity binaries will be available soon.

Docker

Docker is not supported. You can convert your image to a Singularity image, or alternatively try uDocker.

uDocker

Some users have reported successfully running their containers with uDocker.

uDocker is not yet preinstalled.