- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

How to use Conda environments on the clusters: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
(Explains the usage of the conda environment module and the new conda environment builder Docker image)
 
Line 1: Line 1:
This guide shows you how to move conda environments to move the local environment on the clusters without internet access.
This guide shows you how to use Conda environments on the clusters without internet access.


Assumptions:
== Conda Module ==
* You already have a virtual environment called my_env and you want to move this environment.
* The environment can have packages installed with conda and pip.


'''Warning:''' Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment and might compile from source when necessary for the local architecture. '''These packages will run differently for the target system.'''
Miniconda is available as Module, and can be used with packages from main and r channels.


== Using conda-pack ==
'''Warning:''' Miniconda itself is distributed under the 3-clause BSD License, but it allows users to install third-party software with proprietary licenses. You will have to explicitly accept this license when using Miniconda for the first time. Please read carefully the license and third-party licenses which are mentioned there (including Nvidia cuDNN license).


Install <code>conda-pack</code> in the base or root environment:
The first time use is slightly different on Vulcan and Hawk. (''TL;DR: call <code>module load bigdata/conda</code>, and follow instructions'')


<source lang="bash">
<source lang="bash">
(my_env) $ conda deactivate
module load bigdata/conda
(base) $ conda install -c conda-forge conda-pack
source activate # activates the base environment
</source>
</source>
=== Conda Environments ===


Package the environment and transfer the archive to the clusters (e.g., scp):
By default, Conda will activate the default (read-only) base environment with a minimal set of packages. Use Conda as usual. Only main and r channels are available.


<source lang="bash">
'''Warning:''' Conda will create environments in ~/.conda/envs and caches packages under ~/.conda/pkgs. This folder can become quite big and exhaust your quota.
(base) $ conda pack -n my_env -o my_env.tar.gz
</source>


=== Transfer files to a workspace ===
Here is an example:
 
A [[Workspace_mechanism|worskpace]] can be used to refer to the compressed Conda environment. You can send your data to an existing workspace using:


<source lang="bash">
<source lang="bash">
scp <file> <destination_host>:<destination_directory>
module load bigdata/conda
source activate
conda create -n myenv
rm -r ~/.conda/pkgs # delete cache
conda env list
conda activate myenv # you need to execute `source activate` first, or use `source [ENV_PATH]/bin/activate`
</source>
</source>
Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk, described in the next section.


The <code><destination_host></code> can also be replaced with a pre-configured SSH host.
== Custom Conda Environments ==
 
=== Work interactively on a single node ===
 
A large number of files decreases the performance of the parallel file system. You can use the ram disk instead:
 
<source lang="bash">
qsub -I -l select=1:node_type=clx-25 -l walltime=00:30:00 # modify this line to work on Hawk, or to select different resources
 
export ENV_PATH=/run/user/$PBS_JOBID/my_env # We use the ram disk to extract the environment packages since a large number of files decreases the performance of the parallel file system.
export ENV_ARCHIVE=/path/to/my_env.tar.gz # Adjust the path.
 
mkdir -p $ENV_PATH
 
tar -xzf $ENV_ARCHIVE -C $ENV_PATH # This line extracts the packages to RAM disk.
 
source $ENV_PATH/bin/activate
 
conda-unpack
 
# Use the environment here.
 
rm -rf $ENV_PATH # It's nice to clean up before you terminate the job.
</source>
 
==== Notes ====
 
* Extracting the environment directly into a Workspace causes reduction in performance, as compared to the RAM disk.
* This guide assumes there is a <code>$ENV_PATH/bin</code> folder when unpacking the Conda environment.
* This was tested on Conda environments created on Linux, using Python version [https://www.python.org/downloads/release/python-379/ 3.7.9].


'''Warning:''' Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment and might compile from source when necessary for the local architecture. '''These packages will run differently for the target system'''.


=== Use the environment on multiple nodes ===
=== Build, Pack, and Transfer a Custom Conda Environment ===


Preparing a batch script to launch a multi-node job would be best. The steps to start a distributed Python application on multiple nodes depend on the third-party library (e.g., Dask, Ray). Independent of the third-party library, if you want to continue using the ram disk to unpack the virtual environment, you must extract the archive on each node separately. Our documentation provides scripts to [[How_to_launch_a_Ray_cluster_on_HLRS_compute_platforms#Prepare_scripts_to_launch_a_Ray_cluster_on_the_compute_nodes|launch a Ray cluster using conda virtual environments]].
Follow the instructions in the [https://code.hlrs.de/SiVeGCS/conda-env-builder Conda environment builder] repository, which includes example YAML files for building test environments.

Latest revision as of 08:13, 11 April 2024

This guide shows you how to use Conda environments on the clusters without internet access.

Conda Module

Miniconda is available as Module, and can be used with packages from main and r channels.

Warning: Miniconda itself is distributed under the 3-clause BSD License, but it allows users to install third-party software with proprietary licenses. You will have to explicitly accept this license when using Miniconda for the first time. Please read carefully the license and third-party licenses which are mentioned there (including Nvidia cuDNN license).

The first time use is slightly different on Vulcan and Hawk. (TL;DR: call module load bigdata/conda, and follow instructions) 

module load bigdata/conda
source activate # activates the base environment

Conda Environments

By default, Conda will activate the default (read-only) base environment with a minimal set of packages. Use Conda as usual. Only main and r channels are available.

Warning: Conda will create environments in ~/.conda/envs and caches packages under ~/.conda/pkgs. This folder can become quite big and exhaust your quota.

Here is an example: 

module load bigdata/conda
source activate
conda create -n myenv
rm -r ~/.conda/pkgs # delete cache
conda env list
conda activate myenv # you need to execute `source activate` first, or use `source [ENV_PATH]/bin/activate`

Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk, described in the next section.

Custom Conda Environments

Warning: Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment and might compile from source when necessary for the local architecture. These packages will run differently for the target system.

Build, Pack, and Transfer a Custom Conda Environment

Follow the instructions in the Conda environment builder repository, which includes example YAML files for building test environments.

Retrieved from "https://kb.hlrs.de/platforms/index.php?title=How_to_use_Conda_environments_on_the_clusters&oldid=7318"