View source for Urika CS

= Urika-CS nodes =

Urika-CS containers can only be used on CS-Storm (<code>clx-ai</code>) and CS500 (<code>clx-21</code>) nodes.

To be able to start singularity container, the nodes have to be prepared before your batch job starts. You can do this by a resource request ('UNS=True') at your batch submission:

<source lang="bash">qsub -l select=1:node_type=clx-21:UNS=True,walltime=00:30:00 <mybatchjob></source>
Module <code>singularity/3.5.2-cs-patch</code> has some hard-coded settings for the Urika-CS stack.

Urika-CS User Manual for installed versions can be found at Cray website: [https://pubs.cray.com/bundle/CS_Series_Urika-CS_AI_and_Analytics_Applications_Guide_1_2_Nike/page/About_the_CS_Series_Urika-CS_AI_and_Analytics_Applications_Guide.html v1.2UP00], [https://pubs.cray.com/bundle/CS_Series_Urika-CS_AI_and_Analytics_Applications_Guide_1_3_Nike/page/About_the_CS_Series_Urika-CS_AI_and_Analytics_Applications_Guide.html v1.3UP00]. Default version is currently 1.4UP00.

The only module needed to be loaded is: <code>bigdata/analytics</code> - it will load all dependencies and wrappers.

= Quick Start =

== Install cuDNN ==

Due to license limitations you have to install cuDNN by yourself if you accept cuDNN license.

<ol style="list-style-type: decimal;">
<li>Download ( https://developer.nvidia.com/cudnn ) NVIDIA cuDNN for CUDA 10.0, tgz version</li>
<li>Check License you accepted during download</li>
<li>Check License in the downloaded file</li>
<li>If you accept them, upload the archive to login node</li>
<li><p>Extract the archive</p>
<source lang="bash">mkdir -p ~/cudnn/7.6.5.32/cuda10.0
tar -xzf ~/cudnn-10.0-linux-x64-v7.6.5.32.tgz --strip-components=1 -C ~/cudnn/7.6.5.32/cuda10.0</source></li>
<li><p>Create a link to your cuDNN version</p>
<source lang="bash">ln -s ~/cudnn/7.6.5.32/cuda10.0 ~/cudnn/default</source></li></ol>

== Reserve a node (interactive GPU job in this example) ==

<source lang="bash">qsub -I -l select=1:node_type=clx-ai:UNS=true -l walltime=01:00:00</source>
== Load environment ==

<source lang="bash">module load bigdata/analytics</source>
== Start interactive Urika-CS container (this will also deploy Spark) ==

<source lang="bash">start_analytics --cudnn-libs ~/cudnn/default/lib64/

# Activate Conda environment with TensorFlow (GPU)
Singularity> source activate py37_tf2.1.1_gpu

# In the latest Urika-CS image, TensorFlow is installed outside of Conda environment.
# To use TensorFlow set up PYTHONPATH with a path for corresponding version
# Use one of the commands below
## either
export PYTHONPATH=/opt/tensorflow_cpu:$PYTHONPATH
## or
export PYTHONPATH=/opt/tensorflow_gpu:$PYTHONPATH
## or
export PYTHONPATH=/opt/tensorflow_2.1.1/tensorflow_cpu:$PYTHONPATH
## or
export PYTHONPATH=/opt/tensorflow_2.1.1/tensorflow_gpu:$PYTHONPATH
</source>

== Run Urika-CS container in batch mode ==

<source lang="bash">run_training --no-node-list --cudnn-libs ~/cudnn/default/lib64/ -n 1 -e py37_tf_gpu --tensorflow-version 2.1.1 "python test.py"</source>
<code>run_training</code> here has following arguments

* <code>--no-node-list</code> - do not pass node list to CMD
* <code>-n 1</code> - number of MPI processes, usually should correspond to the amount of reserved nodes (<code>run_training</code> executes: <code>mpirun singularity ...</code>)
* <code>-e py37_tf_gpu</code> - this conda environment will be activated in the container
* <code>--tensorflow-version 2.1.1</code> - Specify TensorFlow version (optional). Default is version 1.x (e.g. <code>1.15.2</code>)