View source for How to use Conda environments on the clusters

This guide shows you how to use Conda environments on the clusters without internet access.

== Conda Module ==

Miniconda is available as Module, and can be used with packages from main and r channels.

'''Warning:''' Miniconda itself is distributed under the 3-clause BSD License, but it allows users to install third-party software with proprietary licenses. You will have to explicitly accept this license when using Miniconda for the first time. Please read carefully the license and third-party licenses which are mentioned there (including Nvidia cuDNN license).

The first time use is slightly different on Vulcan and Hawk. (''TL;DR: call <code>module load bigdata/conda</code>, and follow instructions'')

<source lang="bash">
module load bigdata/conda
source activate # activates the base environment
</source>
=== Conda Environments ===

By default, Conda will activate the default (read-only) base environment with a minimal set of packages. Use Conda as usual. Only main and r channels are available.

'''Warning:''' Conda will create environments in ~/.conda/envs and caches packages under ~/.conda/pkgs. This folder can become quite big and exhaust your quota.

Here is an example:

<source lang="bash">
module load bigdata/conda
source activate
conda create -n myenv
rm -r ~/.conda/pkgs # delete cache
conda env list
conda activate myenv # you need to execute `source activate` first, or use `source [ENV_PATH]/bin/activate`
</source>
Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk, described in the next section.

== Custom Conda Environments ==

'''Warning:''' Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment and might compile from source when necessary for the local architecture. '''These packages will run differently for the target system'''.

=== Build, Pack, and Transfer a Custom Conda Environment ===

Follow the instructions in the [https://code.hlrs.de/SiVeGCS/conda-env-builder Conda environment builder] repository, which includes example YAML files for building test environments.