CRAY XC40 Hardware and Architecture

Hazelhen production system

Summary Hazelhen Production system

Cray Cascade XC40 Supercomputer	Step 2
Performance Peak HPL	3.79 Pflops 2.76 Pflops
Cray Cascade Cabinets	21
Number of Compute Nodes	3944 (dual socket)
Compute Processors Total number of CPUs Total number of Cores	3944*2= 7888 Intel Haswell E5-2680v3 2,5 GHz, 12 Cores, 2 HT/Core 7888*12= 94656
Compute Memory on Scalar Processors Memory Type Memory per Compute Node Total Scalar Compute Memory	DDR4 128GB 504832GB= 505TB
Interconnect	Cray Aries
Service Nodes (I/O and Network)	44
External Login Servers	5
Pre- and Post-Processing Servers	3 Cray CS300: each with 4x Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz (Ivy Bridge), 32 cores, 512 GB DDR3 Memory (PC3-14900R), 7,1TB scratch disk space (4x ~2TB RAID0), NVidia Quadro K6000 (12 GB GDDR5), single job usage 2 Cray CS300: each with 4x Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz (Ivy Bridge), 32 cores, 1536 GB DDR3 Memory (PC3-14900R), 15 TB scratch disk space (4x ~4TB RAID0), NVidia Quadro K6000 (12 GB GDDR5), multi job usage
User Storage Lustre Workspace Capacity	5.4PB
Cray Linux Environment (CLE) Compute Node Linux Cluster Compatibility Mode (CCM) Data Virtualization Services (DVS)	Yes
PGI Compiling Suite (FORTRAN, C, C++) including Accelerator	25 user (shared with Step 1)
Cray Developer Toolkit Cray Message Passing Toolkit (MPI, SHMEM, PMI, DMAPP, Global Arrays) PAPI GNU compiler and libraries JAVA Environment setup (Modules) Cray Debugging Support Tools Lgdb STAT ATP	Unlimited Users
Cray Programming Environment Cray Compiling Environment (FORTRAN, C, C++) Cray Performance Monitoring and Analysis Cray PAT Cray Apprentice2 Cray Math and Scientific Libraries Cray Optimized BLAS Cray Optimized LAPACK Cray Optimized ScaLAPACK IRT (Iterative Refinement Toolkit)	Unlimited Users
Alinea DDT Debugger	2048 Processes (shared with Step 1)
Lustre Parallel Filesystem	Licensed on all Sockets
Intel Composer XE Intel C++ Compiler XE Intel Fortran Compiler XE Intel Parallel Debugger Extension Intel Integrated Performance Primitives Intel Cilk Plus Intel Parallel Building Blocks Intel Threading Building Blocks Intel Math Kernel Library	10 Seats

For detailed information see XC40-Intro

Architecture

• System Management Workstation (SMW)
  • system administrator's console for managing a Cray system like monitoring, installing/upgrading software, controls the hardware, starting and stopping the XC40 system.

• service nodes are classified in:
  • login nodes for users to access the system
  • boot nodes which provides the OS for all other nodes, licenses,...
  • network nodes which provides e.g. external network connections for the compute nodes
  • Cray Data Virtualization Service (DVS): is an I/O forwarding service that can parallelize the I/O transactions of an underlying POSIX-compliant file system.
  • sdb node for services like ALPS, torque, moab, slurm, cray management services,...
  • I/O nodes for e.g. lustre
  • MOM nodes for placing user jobs of the batch system in to execution

• compute nodes
  • are only available for user using the batch system and the Application Level Placement Scheduler (ALPS), see running applications.
    • The compute nodes are installed with 64 GB memory, each with fast interconnect (CRAY Aries).
    • Details about the interconnect of the Cray XC series network

• in future, the StorageSwitch Fabric of step2a and step1 will be connected. So, the Lustre workspace filesystems can be used on both hardware (Login servers and preprocessing servers) of step1 and step2a.