How to use Conda environments on the clusters

This guide shows you how to use Conda environments on the clusters without internet access.

Conda Module

Miniconda is available as Module, and can be used with packages from main and r channels.

Warning: Miniconda itself is distributed under the 3-clause BSD License, but it allows users to install third-party software with proprietary licenses. You will have to explicitly accept this license when using Miniconda for the first time. Please read carefully the license and third-party licenses which are mentioned there (including Nvidia cuDNN license).

The first time use is slightly different on Vulcan and Hawk. (TL;DR: call module load bigdata/conda, and follow instructions)

module load bigdata/conda
source activate # activates the base environment

Conda Environments

By default, Conda will activate the default (read-only) base environment with a minimal set of packages. Use Conda as usual. Only main and r channels are available.

Warning: Conda will create environments in ~/.conda/envs and caches packages under ~/.conda/pkgs. This folder can become quite big and exhaust your quota.

Here is an example:

module load bigdata/conda
source activate
conda create -n myenv
rm -r ~/.conda/pkgs # delete cache
conda env list
conda activate myenv # you need to execute `source activate` first, or use `source [ENV_PATH]/bin/activate`

Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk, described in the next section.

Custom Conda Environments

Warning: Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment and might compile from source when necessary for the local architecture. These packages will run differently for the target system.

Build, Pack, and Transfer a Custom Conda Environment

Follow the instructions in the Conda environment builder repository, which includes example YAML files for building test environments.