- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

NEC Cluster Hardware and Architecture (vulcan): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
(Created page with "__TOC__ === Hardware === <s> * ~150 compute nodes are of type NEC HPC-144 Rb-1 Server (see [http://www.nec.com/de/prod/solutions/lx-series/index.html NEC Products]) ** dual C...")
 
No edit summary
 
(16 intermediate revisions by 5 users not shown)
Line 1: Line 1:
__TOC__
__TOC__
=== Hardware ===
=== Hardware ===
<s>
* ~150 compute nodes are of type NEC HPC-144 Rb-1 Server (see [http://www.nec.com/de/prod/solutions/lx-series/index.html  NEC Products])
** dual CPU compute nodes: 2x [http://ark.intel.com/Product.aspx?id=37109&processor=X5560&spec-codes=SLBF4 Intel Xeon X5560] Nehalem EP "Gainestown" ([http://www.intel.com/products/processor/xeon5000/ 5000 Sequence] [http://www.intel.com/products/processor/xeon5000/specifications.htm specifications])
*** 4 cores, 8 threads
*** 2.80 GHz (3.20 Ghz max. Turbo frequency)
*** 8MB L3 Cache
*** 1333 MHz Memory Interface, 6.4 GT/s QPI
*** TDP 95W, 45nm technology
*** [http://de.wikipedia.org/wiki/Intel-Nehalem-Mikroarchitektur "Nehalem"] microarchitecture
** compute node RAM: triple-channel memory
*** standard: 12 GB RAM (''nehalem''/''mem12gb'')
*** 20 nodes upgraded to 24GB (''mem24gb''), 48GB (''mem48gb'') or 144GB (''mem144gb'') RAM
**** 2 nodes of 144GB Memory nodes have additional a 6TB local scratch disk installed
**** 1 nodes of 144GB Memory nodes have additional a 2TB local scratch disk installed
** 16 compute nodes have additional [http://www.nvidia.com/object/tesla_computing_solutions.html Nvidia Tesla S1070 GPU's] installed.
</s>


*''' Pre- & Postprocessing node''' (''smp'' node)
** 8x Intel Xeon [http://ark.intel.com/products/46497/Intel-Xeon-Processor-X7542-(18M-Cache-2_66-GHz-5_86-GTs-Intel-QPI) X7542] 6-core CPUs with 2.67GHz (8*6=48 Cores)
** 1TB RAM
** shared access


*'''Visualisation node''' (''vis'')
* ''' Pre- & Postprocessing node''' (''smp'' node) [Entgeltordnung: 'smp Knoten']
** 5 nodes each with 8 cores Intel [http://ark.intel.com/de/products/39719/Intel-Xeon-Processor-W3540-8M-Cache-2_93-GHz-4_80-GTs-Intel-QPI W3540] and 24GB memory (4 for laki and 1 for laki2)
** 1 node
*** Nvidia Quadro FX5800
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/120489/intel-xeon-gold-6148-processor-27-5m-cache-2-40-ghz.html Xeon Gold 6148], 40 cores total @ 2.40GHz
*** 1.5TB memory
*** Nvidia Quadro K6000
*** 2TB HDD mounted on /tmp
*** 13TB HDD mounted on /localscratch
*** shared access


* '''Node Upgrades''' (2012/2013)
* '''CascadeLake 40 cores compute nodes''' (''clx'')  
** 128 nodes Dual Intel [[Sb|'Sandy Bridge']] [http://ark.intel.com/de/products/64595/Intel-Xeon-Processor-E5-2670-20M-Cache-2_60-GHz-8_00-GTs-Intel-QPI E5-2670] (204 for laki and 124 for laki2)
** 96 nodes (''clx-25'', ''clx384gb40c'') [Entgeltordnung: 'Cascadelake 384 GB']
*** 2.6 Ghz, 8 Cores per processor, 16 Threads
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz.html Xeon Gold 6248], 40 cores total @ 2.50GHz
*** 4 memory channels per processor, DDR3 1600Mhz memory
*** 384GB memory
*** 96 nodes with 32GB RAM (''sb''/''mem32gb'')
** 8 nodes (''clx-21'', ''clx384gb40c-ai'') [Entgeltordnung: 'CS500-Spark']
*** 30 nodes with 64GB RAM (''mem64gb'')
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz.html Xeon Gold 6230], 40 cores total @ 2.10GHz
*** QDR Mellanox ConnectX-3 IB HCAs (40gbit)
*** 384GB memory
*** 1.8TB NVMe mounted at /localscratch


* '''Node Upgrades''' (2014/2015)
* '''CascadeLake 36 cores compute nodes''' (''clx-ai'') for artificial intelligence and big data applications [Entgeltordnung: 'CS Storm']
** 80 nodes Dual Intel [[hsw|'Haswell']] [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz E5-2660v3]
** 4 nodes (''clx768gb36c-ai'')
*** 2.6 Ghz, 10 Cores per processor, 20 Threads
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html Xeon Gold 6240], 36 cores total @ 2.60GHz
*** 4 memory channels per processor, DDR4 2133Mhz memory
*** 768GB memory
*** 76 nodes with 128GB RAM (''hsw128gb10c'')
*** 8x Nvidia Tesla V100 SXM2 32GB
*** 4 nodes with 256GB RAM (''hsw256gb10c'')
*** 7.3TB NVMe mounted at /localscratch
*** QDR Mellanox ConnectX-3 IB HCAs (40gbit)
*** 220GB SSD mounted at /tmp


* '''Node Upgrades''' (2016/17)
* '''Haswell 20 Cores compute nodes''' (''hsw'')
** 360 nodes Dual Intel [[hsw|'Haswell']] [http://ark.intel.com/de/products/81908/Intel-Xeon-Processor-E5-2680-v3-30M-Cache-2_50-GHz E5-2680v3]
** 2x Intel [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz Xeon E5-2660v3], 20 cores total @ 2.60GHz
*** 2.5 Ghz, 12 Cores per processor, 24 Threads
** 76 nodes (''hsw128gb20c'') [Entgeltordnung: 'Haswell 128 GB']
*** 4 memory channels per processor, DDR4 2133Mhz memory
*** 128GB RAM
*** 344 nodes with 128GB RAM (''hsw128gb12c'')  
** 4 nodes (''hsw256gb20c'') [Entgeltordnung: 'Haswell 256 GB']
*** 16 nodes with 256GB RAM (''hsw256gb12c'')
*** 256GB RAM
*** QDR Mellanox ConnectX-3 IB HCAs (40gbit), 144 of the 128GB nodes have fdr IB, (''fdr'')


* '''Additional [[Mem256gb|large memory nodes]]'''
* '''Haswell 24 Cores compute nodes''' (''hsw'')
** 10 nodes Quad Socket AMD Opteron [http://products.amd.com/en-ca/search/CPU/AMD-Opteron%E2%84%A2/AMD-Opteron%E2%84%A2-6200-Series-Processor/6238/32 6238]
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz.html Xeon E5-2668v3], 24 cores total @ 2.50GHz
** 2.6 Ghz, 12 cores per processor
** 152 nodes (''hsw128gb24c'') [Entgeltordnung: 'Haswell 128 GB']
** 4 memory channels per processor, DDR3 1600Mhz memory
*** 128GB memory
** 256GB RAM (''mem256gb'')
** 16 nodes (''hsw256gb24c'') [Entgeltordnung: 'Haswell 256 GB']
** QDR Mellanox ConnectX-2 IB HCAs (40gbit)
*** 256GB memory
** 4 nodes have additional a 4TB local scratch disk


* '''(Nov. 2017) 10 Additional GPU graphic nodes with'''
* '''Skylake 40 Cores compute nodes''' (''skl'') [Entgeltordnung: 'SKL-192GB']
** Nvidia Tesla P100 12GB
** 88 (''skl192gb40c'')
** 2 sockets ech 8 cores (Intel E5-2667v4 @ 3.2GHz)
*** 2x Intel [https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/gold-processors/gold-6138.html Xeon Gold 6138], 40 cores total | 2.00GHz
** 256GB memory
*** 192GB memory
** 3.7TB /localscratch, 400GB SSD /tmp


*'''Visualisation node with GPUs'''
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 codes total @ 2.60GHz
** 96GB memory
** 6 nodes (''visamd'') [Entgeltordnung: 'Graphikserver AMD']
*** AMD Radeon Pro WX8200
** 1 node (''visnv'') [Entgeltordnung: 'Graphikserver NVIDIA']
*** Nvidia Quadro RTX 4000


* '''network''': [http://de.wikipedia.org/wiki/Infiniband InfiniBand] Double Data Rate
* '''Visualisation/GPGPU graphic nodes''' (''visp100'') [Entgeltordnung: 'VISP 100']
** switches for interconnect: [http://www.voltaire.com/Products/Grid_Backbone_Switches Voltaire Grid Director] [http://www.voltaire.com/Products/InfiniBand/Grid_Director_Switches/Voltaire_Grid_Director_4036 4036] with 36 QDR (40Gbps) ports (6 backbone switches)
** 10 nodes
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/92979/intel-xeon-processor-e5-2667-v4-25m-cache-3-20-ghz.html Xeon E5-2667v4], 16 cores total @ 3.20GHz
*** 256GB memory
*** Nvidia Tesla P100 12GB
*** 3.7TB SSD mounted at /localscratch
*** 400GB SSD mounted at /tmp




 
* '''Interconnect''': [http://de.wikipedia.org/wiki/Infiniband InfiniBand]
 
** Various generations of Infiniband switches with QDR, FDR, EDR and HDR speed
 
* [http://www.top500.org/ Top500] rankings for system [http://www.top500.org/system/9888 Baku]:
** [http://www.top500.org/lists/2009/06 June 2009 list] [http://www.top500.org/list/2009/06/100 (1-100)]: #77
** [http://www.top500.org/lists/2009/11 November 2009 list] [http://www.top500.org/list/2009/11/100 (1-100)]: #94
** [http://www.top500.org/lists/2010/06 June 2010 list] [http://www.top500.org/list/2010/06/200 (101-200)]: #110
** [http://www.top500.org/lists/2010/11 November 2010 list] [http://www.top500.org/list/2010/11/200 (101-200)]: #157
** [http://www.top500.org/lists/2011/06 June 2011 list] [http://www.top500.org/list/2011/06/400 (301-400)]: #305
 
* [http://www.green500.org/ Green500] rankings:
** [http://www.green500.org/lists/2009/06/top/list.php June 2009 list] [http://www.green500.org/lists/2009/06/top/list.php?from=1&to=100 (1-100)]: [http://www.green500.org/cert1.php?list=green201006&green500_rank=20 #20]
** [http://www.green500.org/lists/2009/11/top/list.php November 2009 list] [http://www.green500.org/lists/2009/11/top/list.php?from=1&to=100 (1-100)]: [http://www.green500.org/cert1.php?list=green200911&green500_rank=30 #30]
** [http://www.green500.org/lists/2010/06/top/list.php June 2010 list] [http://www.green500.org/lists/2010/06/top/list.php?from=1&to=100 (1-100)]: [http://www.green500.org/cert1.php?list=green201006&green500_rank=48 #48]
** [http://www.green500.org/lists/2010/11/top/list.php November 2010 list] [http://www.green500.org/lists/2010/11/top/list.php?from=1&to=100 (1-100)]: [http://www.green500.org/cert1.php?list=green201011&green500_rank=72 #72]
** [http://www.green500.org/lists/2011/06/top/list.php June 2011 list] [http://www.green500.org/lists/2011/06/top/list.php?from=1&to=100 (1-100)]: [http://www.green500.org/cert1.php?list=green201106&green500_rank=86 #86]


=== Architecture ===
=== Architecture ===


The NEC Cluster platform (vulcan) consists of several '''frontend nodes''' for interactive access (for access details see [[NEC_Cluster_access_(vulcan)| Access]]) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.  
The NEC Cluster platform (vulcan) consists of several '''frontend nodes''' for interactive access (for access details see [[NEC_Cluster_access_(vulcan)| Access]]) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.  


'''Compute node types installed:'''  
'''Compute node types installed:'''  
<s> * Intel Xeon 5560 (nehalem) </s>
* Haswell, Skylake, CascadeLake
* Intel Xeon E5-2670 (Sandy Bridge)
* different Memory nodes (128GB, 256GB, 384GB)
* AMD Opteron 6238 (interlagos)
* Pre-Postprocessing node with very large memory (1.5TB)
* Intel E5-2680v3 and  E5-2660v3
* Visualisation/GPU nodes with AMD Radeon Pro WX8200, Nvidia Quadro RTX4000 or Nvidia Tesla P100
<s> * Nvidia Tesla S1070 (consisting of C1060 devices) </s>
* KI nodes with Nvidia Tesla V100
* Large Memory nodes (144GB, 256GB)
* Vector nodes with NEC Aurora TSUBASA CPUs
* Pre-Postprocessing node with very large memory (1TB)
* Visualisation nodes with Nvidia Quadro FX5800 or Nvidia Tesla P100
* Different memory nodes (<s> 12GB, 24GB,</s> 32GB, <s>48GB</s>, 64GB, 128GB, 256GB)
 
    
    
'''Features'''
'''Features'''
Line 108: Line 84:
* Batchsystem: PBSPro
* Batchsystem: PBSPro
* node-node interconnect: Infiniband + GigE
* node-node interconnect: Infiniband + GigE
* Global Disk 500 TB (lustre) for vulcan + 500TB (lustre) for vulcan2
* Global Disk 2.2 PB (lustre) for vulcan + 500TB (lustre) for vulcan2
* Many Software Packages for Development
* Many Software Packages for Development

Latest revision as of 10:58, 4 October 2023

Hardware

  • Pre- & Postprocessing node (smp node) [Entgeltordnung: 'smp Knoten']
    • 1 node
      • 2x Intel Xeon Gold 6148, 40 cores total @ 2.40GHz
      • 1.5TB memory
      • Nvidia Quadro K6000
      • 2TB HDD mounted on /tmp
      • 13TB HDD mounted on /localscratch
      • shared access
  • CascadeLake 40 cores compute nodes (clx)
    • 96 nodes (clx-25, clx384gb40c) [Entgeltordnung: 'Cascadelake 384 GB']
    • 8 nodes (clx-21, clx384gb40c-ai) [Entgeltordnung: 'CS500-Spark']
      • 2x Intel Xeon Gold 6230, 40 cores total @ 2.10GHz
      • 384GB memory
      • 1.8TB NVMe mounted at /localscratch
  • CascadeLake 36 cores compute nodes (clx-ai) for artificial intelligence and big data applications [Entgeltordnung: 'CS Storm']
    • 4 nodes (clx768gb36c-ai)
      • 2x Intel Xeon Gold 6240, 36 cores total @ 2.60GHz
      • 768GB memory
      • 8x Nvidia Tesla V100 SXM2 32GB
      • 7.3TB NVMe mounted at /localscratch
      • 220GB SSD mounted at /tmp
  • Haswell 20 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2660v3, 20 cores total @ 2.60GHz
    • 76 nodes (hsw128gb20c) [Entgeltordnung: 'Haswell 128 GB']
      • 128GB RAM
    • 4 nodes (hsw256gb20c) [Entgeltordnung: 'Haswell 256 GB']
      • 256GB RAM
  • Haswell 24 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2668v3, 24 cores total @ 2.50GHz
    • 152 nodes (hsw128gb24c) [Entgeltordnung: 'Haswell 128 GB']
      • 128GB memory
    • 16 nodes (hsw256gb24c) [Entgeltordnung: 'Haswell 256 GB']
      • 256GB memory
  • Skylake 40 Cores compute nodes (skl) [Entgeltordnung: 'SKL-192GB']
    • 88 (skl192gb40c)
  • Visualisation node with GPUs
    • 2x Intel Xeon Silver 4112, 8 codes total @ 2.60GHz
    • 96GB memory
    • 6 nodes (visamd) [Entgeltordnung: 'Graphikserver AMD']
      • AMD Radeon Pro WX8200
    • 1 node (visnv) [Entgeltordnung: 'Graphikserver NVIDIA']
      • Nvidia Quadro RTX 4000
  • Visualisation/GPGPU graphic nodes (visp100) [Entgeltordnung: 'VISP 100']
    • 10 nodes
      • 2x Intel Xeon E5-2667v4, 16 cores total @ 3.20GHz
      • 256GB memory
      • Nvidia Tesla P100 12GB
      • 3.7TB SSD mounted at /localscratch
      • 400GB SSD mounted at /tmp


  • Interconnect: InfiniBand
    • Various generations of Infiniband switches with QDR, FDR, EDR and HDR speed

Architecture

The NEC Cluster platform (vulcan) consists of several frontend nodes for interactive access (for access details see Access) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.

Compute node types installed:

  • Haswell, Skylake, CascadeLake
  • different Memory nodes (128GB, 256GB, 384GB)
  • Pre-Postprocessing node with very large memory (1.5TB)
  • Visualisation/GPU nodes with AMD Radeon Pro WX8200, Nvidia Quadro RTX4000 or Nvidia Tesla P100
  • KI nodes with Nvidia Tesla V100
  • Vector nodes with NEC Aurora TSUBASA CPUs

Features

  • Operating System: Centos 7
  • Batchsystem: PBSPro
  • node-node interconnect: Infiniband + GigE
  • Global Disk 2.2 PB (lustre) for vulcan + 500TB (lustre) for vulcan2
  • Many Software Packages for Development