- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

NEC Cluster Hardware and Architecture (vulcan): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
 
(6 intermediate revisions by 3 users not shown)
Line 3: Line 3:




* ''' Pre- & Postprocessing node''' (''smp'' node)
* ''' Pre- & Postprocessing node''' (''smp'' node) [Entgeltordnung: 'smp Knoten']
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/120489/intel-xeon-gold-6148-processor-27-5m-cache-2-40-ghz.html Xeon Gold 6148], 40 cores total @ 2.40GHz
** 1 node
** 1.5TB memory
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/120489/intel-xeon-gold-6148-processor-27-5m-cache-2-40-ghz.html Xeon Gold 6148], 40 cores total @ 2.40GHz
** Nvidia Quadro K6000
*** 1.5TB memory
** 2TB HDD mounted on /tmp
*** Nvidia Quadro K6000
** 13TB HDD mounted on /localscratch
*** 2TB HDD mounted on /tmp
** shared access
*** 13TB HDD mounted on /localscratch
*** shared access


* '''CascadeLake 40 cores compute nodes''' (''clx'')
* '''CascadeLake 40 cores compute nodes''' (''clx'')  
** 96 nodes (''clx-25'', ''clx384gb40c'')
** 96 nodes (''clx-25'', ''clx384gb40c'') [Entgeltordnung: 'Cascadelake 384 GB']
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz.html Xeon Gold 6248], 40 cores total @ 2.50GHz
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz.html Xeon Gold 6248], 40 cores total @ 2.50GHz
*** 384GB memory
*** 384GB memory
** 8 nodes (''clx-21'', ''clx384gb40c-ai'')
** 8 nodes (''clx-21'', ''clx384gb40c-ai'') [Entgeltordnung: 'CS500-Spark']
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz.html Xeon Gold 6230], 40 cores total @ 2.10GHz
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz.html Xeon Gold 6230], 40 cores total @ 2.10GHz
*** 384GB memory
*** 384GB memory
*** 1.8TB NVMe mounted at /localscratch
*** 1.8TB NVMe mounted at /localscratch


* '''CascadeLake 36 cores compute nodes''' (''clx-ai'') for artificial intelligence and big data applications
* '''CascadeLake 36 cores compute nodes''' (''clx-ai'') for artificial intelligence and big data applications [Entgeltordnung: 'CS Storm']
** 8 nodes (''clx768gb36c-ai'')
** 4 nodes (''clx768gb36c-ai'')
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html Xeon Gold 6240], 36 cores total @ 2.60GHz
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html Xeon Gold 6240], 36 cores total @ 2.60GHz
*** 768GB memory
*** 768GB memory
Line 30: Line 31:
* '''Haswell 20 Cores compute nodes''' (''hsw'')
* '''Haswell 20 Cores compute nodes''' (''hsw'')
** 2x Intel [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz Xeon E5-2660v3], 20 cores total @ 2.60GHz
** 2x Intel [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz Xeon E5-2660v3], 20 cores total @ 2.60GHz
** 84 nodes (''hsw128gb20c'')
** 76 nodes (''hsw128gb20c'') [Entgeltordnung: 'Haswell 128 GB']
*** 128GB RAM
*** 128GB RAM
** 4 nodes (''hsw256gb20c'')
** 4 nodes (''hsw256gb20c'') [Entgeltordnung: 'Haswell 256 GB']
*** 256GB RAM
*** 256GB RAM


* '''Haswell 24 Cores compute nodes''' (''hsw'')
* '''Haswell 24 Cores compute nodes''' (''hsw'')
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz.html Xeon E5-2668v3], 24 cores total @ 2.50GHz
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz.html Xeon E5-2668v3], 24 cores total @ 2.50GHz
** 152 nodes (''hsw128gb24c'')
** 152 nodes (''hsw128gb24c'') [Entgeltordnung: 'Haswell 128 GB']
*** 128GB memory
*** 128GB memory
** 16 nodes (''hsw256gb24c'')
** 16 nodes (''hsw256gb24c'') [Entgeltordnung: 'Haswell 256 GB']
*** 256GB memory
*** 256GB memory


* '''Skylake 40 Cores compute nodes''' (''skl'')
* '''Skylake 40 Cores compute nodes''' (''skl'') [Entgeltordnung: 'SKL-192GB']
** 100 nodes (''skl192gb40c'')
** 88 (''skl192gb40c'')
*** 2x Intel [https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/gold-processors/gold-6138.html Xeon Gold 6138], 40 cores total | 2.00GHz
*** 2x Intel [https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/gold-processors/gold-6138.html Xeon Gold 6138], 40 cores total | 2.00GHz
*** 192GB memory
*** 192GB memory
*'''Visualisation node with AMD graphics''' (''visamd'')
 
** 6 nodes
*'''Visualisation node with GPUs'''
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 codes total @ 2.60GHz
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 codes total @ 2.60GHz
*** 96GB memory
** 96GB memory
** 6 nodes (''visamd'') [Entgeltordnung: 'Graphikserver AMD']
*** AMD Radeon Pro WX8200
*** AMD Radeon Pro WX8200
 
** 1 node (''visnv'') [Entgeltordnung: 'Graphikserver NVIDIA']
*'''Visualisation node with NVIDIA graphics''' (''visnv'')
** 1 node
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 cores total @ 2.60GHz
*** 96GB memory
*** Nvidia Quadro RTX 4000
*** Nvidia Quadro RTX 4000


* '''Visualisation/GPGPU graphic nodes''' (''visp100'')
* '''Visualisation/GPGPU graphic nodes''' (''visp100'') [Entgeltordnung: 'VISP 100']
** 10 nodes
** 10 nodes
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/92979/intel-xeon-processor-e5-2667-v4-25m-cache-3-20-ghz.html Xeon E5-2667v4], 16 cores total @ 3.20GHz
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/92979/intel-xeon-processor-e5-2667-v4-25m-cache-3-20-ghz.html Xeon E5-2667v4], 16 cores total @ 3.20GHz
Line 73: Line 71:


The NEC Cluster platform (vulcan) consists of several '''frontend nodes''' for interactive access (for access details see [[NEC_Cluster_access_(vulcan)| Access]]) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.  
The NEC Cluster platform (vulcan) consists of several '''frontend nodes''' for interactive access (for access details see [[NEC_Cluster_access_(vulcan)| Access]]) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.  


'''Compute node types installed:'''  
'''Compute node types installed:'''  
* Sandybridge, Haswell, Skylake  
* Haswell, Skylake, CascadeLake
* different Memory nodes (32GB, 64GB, 128GB, 256GB, 384GB)
* different Memory nodes (128GB, 256GB, 384GB)
* Pre-Postprocessing node with very large memory (1TB)
* Pre-Postprocessing node with very large memory (1.5TB)
* Visualisation/GPU nodes with Nvidia Quadro FX5800 or Nvidia Tesla P100
* Visualisation/GPU nodes with AMD Radeon Pro WX8200, Nvidia Quadro RTX4000 or Nvidia Tesla P100
 
* KI nodes with Nvidia Tesla V100
 
* Vector nodes with NEC Aurora TSUBASA CPUs
    
    
'''Features'''
'''Features'''
Line 87: Line 84:
* Batchsystem: PBSPro
* Batchsystem: PBSPro
* node-node interconnect: Infiniband + GigE
* node-node interconnect: Infiniband + GigE
* Global Disk 500 TB (lustre) for vulcan + 500TB (lustre) for vulcan2
* Global Disk 2.2 PB (lustre) for vulcan + 500TB (lustre) for vulcan2
* Many Software Packages for Development
* Many Software Packages for Development

Latest revision as of 10:58, 4 October 2023

Hardware

  • Pre- & Postprocessing node (smp node) [Entgeltordnung: 'smp Knoten']
    • 1 node
      • 2x Intel Xeon Gold 6148, 40 cores total @ 2.40GHz
      • 1.5TB memory
      • Nvidia Quadro K6000
      • 2TB HDD mounted on /tmp
      • 13TB HDD mounted on /localscratch
      • shared access
  • CascadeLake 40 cores compute nodes (clx)
    • 96 nodes (clx-25, clx384gb40c) [Entgeltordnung: 'Cascadelake 384 GB']
    • 8 nodes (clx-21, clx384gb40c-ai) [Entgeltordnung: 'CS500-Spark']
      • 2x Intel Xeon Gold 6230, 40 cores total @ 2.10GHz
      • 384GB memory
      • 1.8TB NVMe mounted at /localscratch
  • CascadeLake 36 cores compute nodes (clx-ai) for artificial intelligence and big data applications [Entgeltordnung: 'CS Storm']
    • 4 nodes (clx768gb36c-ai)
      • 2x Intel Xeon Gold 6240, 36 cores total @ 2.60GHz
      • 768GB memory
      • 8x Nvidia Tesla V100 SXM2 32GB
      • 7.3TB NVMe mounted at /localscratch
      • 220GB SSD mounted at /tmp
  • Haswell 20 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2660v3, 20 cores total @ 2.60GHz
    • 76 nodes (hsw128gb20c) [Entgeltordnung: 'Haswell 128 GB']
      • 128GB RAM
    • 4 nodes (hsw256gb20c) [Entgeltordnung: 'Haswell 256 GB']
      • 256GB RAM
  • Haswell 24 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2668v3, 24 cores total @ 2.50GHz
    • 152 nodes (hsw128gb24c) [Entgeltordnung: 'Haswell 128 GB']
      • 128GB memory
    • 16 nodes (hsw256gb24c) [Entgeltordnung: 'Haswell 256 GB']
      • 256GB memory
  • Skylake 40 Cores compute nodes (skl) [Entgeltordnung: 'SKL-192GB']
    • 88 (skl192gb40c)
  • Visualisation node with GPUs
    • 2x Intel Xeon Silver 4112, 8 codes total @ 2.60GHz
    • 96GB memory
    • 6 nodes (visamd) [Entgeltordnung: 'Graphikserver AMD']
      • AMD Radeon Pro WX8200
    • 1 node (visnv) [Entgeltordnung: 'Graphikserver NVIDIA']
      • Nvidia Quadro RTX 4000
  • Visualisation/GPGPU graphic nodes (visp100) [Entgeltordnung: 'VISP 100']
    • 10 nodes
      • 2x Intel Xeon E5-2667v4, 16 cores total @ 3.20GHz
      • 256GB memory
      • Nvidia Tesla P100 12GB
      • 3.7TB SSD mounted at /localscratch
      • 400GB SSD mounted at /tmp


  • Interconnect: InfiniBand
    • Various generations of Infiniband switches with QDR, FDR, EDR and HDR speed

Architecture

The NEC Cluster platform (vulcan) consists of several frontend nodes for interactive access (for access details see Access) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.

Compute node types installed:

  • Haswell, Skylake, CascadeLake
  • different Memory nodes (128GB, 256GB, 384GB)
  • Pre-Postprocessing node with very large memory (1.5TB)
  • Visualisation/GPU nodes with AMD Radeon Pro WX8200, Nvidia Quadro RTX4000 or Nvidia Tesla P100
  • KI nodes with Nvidia Tesla V100
  • Vector nodes with NEC Aurora TSUBASA CPUs

Features

  • Operating System: Centos 7
  • Batchsystem: PBSPro
  • node-node interconnect: Infiniband + GigE
  • Global Disk 2.2 PB (lustre) for vulcan + 500TB (lustre) for vulcan2
  • Many Software Packages for Development
Retrieved from "https://kb.hlrs.de/platforms/index.php?title=NEC_Cluster_Hardware_and_Architecture_(vulcan)&oldid=7176"