- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

NEC Cluster Hardware and Architecture (vulcan): Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
Line 3: Line 3:




*''' Pre- & Postprocessing node''' (''smp'' node)
* ''' Pre- & Postprocessing node''' (''smp'' node)
** 8x Intel Xeon [http://ark.intel.com/products/46497/Intel-Xeon-Processor-X7542-(18M-Cache-2_66-GHz-5_86-GTs-Intel-QPI) X7542] 6-core CPUs with 2.67GHz (8*6=48 Cores)
** 8x Intel [http://ark.intel.com/products/46497/Intel-Xeon-Processor-X7542-(18M-Cache-2_66-GHz-5_86-GTs-Intel-QPI) Xeon X7542], 48 cores total @ 2.67GHz
** 1TB RAM
** 1TB memory
** shared access
** shared access


*'''Visualisation node''' (''vis'')
* '''CascadeLake 40 cores compute nodes''' (''clx'')
** ??? nodes each with 8 cores Intel [http://ark.intel.com/de/products/39719/Intel-Xeon-Processor-W3540-8M-Cache-2_93-GHz-4_80-GTs-Intel-QPI W3540] and 24GB memory
** 96 nodes (''clx-25'', ''clx384gb40c'')
*** Nvidia Quadro FX5800
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz.html Xeon Gold 6248], 40 cores total @ 2.50GHz
*** 384GB memory
** 8 nodes (''clx-21'', ''clx384gb40c-ai'')
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz.html Xeon Gold 6230], 40 cores total @ 2.10GHz
*** 384GB memory
*** 1.8TB NVMe mounted at /localscratch


* '''CascadeLake 36 cores compute nodes''' (''clx-ai'') for artificial intelligence and big data applications
** 8 nodes (''clx768gb36c-ai'')
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html Xeon Gold 6240], 36 cores total @ 2.60GHz
*** 768GB memory
*** 8x Nvidia Tesla V100 SXM2 32GB
*** 7.3TB NVMe mounted at /localscratch
*** 220GB SSD mounted at /tmp


* '''Haswell 20 Cores compute nodes'''
* '''Haswell 20 Cores compute nodes''' (''hsw'')
** 80 nodes Dual Intel [[hsw|'Haswell']] [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz E5-2660v3]
** 2x Intel [http://ark.intel.com/de/products/81706/Intel-Xeon-Processor-E5-2660-v3-25M-Cache-2_60-GHz Xeon E5-2660v3], 20 cores total @ 2.60GHz
*** 2.6 Ghz, 10 Cores per processor, 20 Threads
** 84 nodes (''hsw128gb20c'')
*** 4 memory channels per processor, DDR4 2133Mhz memory
*** 128GB RAM
*** 76 nodes with 128GB RAM (''hsw128gb10c'')
** 4 nodes (''hsw256gb20c'')
*** 4 nodes with 256GB RAM (''hsw256gb10c'')
*** 256GB RAM
*** QDR Mellanox ConnectX-3 IB HCAs (40gbit)


* '''Haswell 24 Cores compute nodes'''
* '''Haswell 24 Cores compute nodes''' (''hsw'')
** 168 nodes Dual Intel [[hsw|'Haswell']] [http://ark.intel.com/de/products/81908/Intel-Xeon-Processor-E5-2680-v3-30M-Cache-2_50-GHz E5-2680v3]
** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz.html Xeon E5-2668v3], 24 cores total @ 2.50GHz
*** 2.5 Ghz, 12 Cores per processor, 24 Threads
** 152 nodes (''hsw128gb24c'')
*** 4 memory channels per processor, DDR4 2133Mhz memory
*** 128GB memory
*** 152 nodes with 128GB RAM (''hsw128gb12c'')  
** 16 nodes (''hsw256gb24c'')
*** 16 nodes with 256GB RAM (''hsw256gb12c'')
*** 256GB memory
*** QDR Mellanox ConnectX-3 IB HCAs (40gbit), 144 of the 128GB nodes have fdr IB, (''fdr'')


*'''Skylake 40 Cores compute nodes'''
* '''Skylake 40 Cores compute nodes''' (''skl'')
** 100 nodes Dual Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz [https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/gold-processors/gold-6138.html]
** 100 nodes (''skl192gb40c'')
*** 2.0GHz, 20 Cores per processor, 40 Threads
*** 2x Intel [https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/gold-processors/gold-6138.html Xeon Gold 6138], 40 cores total | 2.00GHz
*** 6 memory channels, DDR4 2666 MHz memory
*** 192GB memory
*** 192 GB RAM ("skl192gb20c")
*'''Visualisation node with AMD graphics''' (''visamd'')
*** EDR Mellanox ConnectX-5 IB HCAs (100gbit)
** 6 nodes
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 codes total @ 2.60GHz
*** 96GB memory
*** AMD Radeon Pro WX8200


* '''10 Visualisation/GPU graphic nodes with'''
*'''Visualisation node with NVIDIA graphics''' (''visnv'')
** Nvidia Tesla P100 12GB
** 1 node
** 2 sockets ech 8 cores (Intel E5-2667v4 @ 3.2GHz)
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html Xeon Silver 4112], 8 cores total @ 2.60GHz
** 256GB memory
*** 96GB memory
** 3.7TB /localscratch, 400GB SSD /tmp
*** Nvidia Quadro RTX 4000


* '''Visualisation/GPGPU graphic nodes''' (''visp100'')
** 10 nodes
*** 2x Intel [https://ark.intel.com/content/www/us/en/ark/products/92979/intel-xeon-processor-e5-2667-v4-25m-cache-3-20-ghz.html Xeon E5-2667v4], 16 cores total @ 3.20GHz
*** 256GB memory
*** Nvidia Tesla P100 12GB
*** 3.7TB SSD mounted at /localscratch
*** 400GB SSD mounted at /tmp


* '''network''': [http://de.wikipedia.org/wiki/Infiniband InfiniBand] Double Data Rate
 
** switches for interconnect: [http://www.voltaire.com/Products/Grid_Backbone_Switches Voltaire Grid Director] [http://www.voltaire.com/Products/InfiniBand/Grid_Director_Switches/Voltaire_Grid_Director_4036 4036] with 36 QDR (40Gbps) ports (6 backbone switches)
* '''Interconnect''': [http://de.wikipedia.org/wiki/Infiniband InfiniBand]
** Various generations of Infiniband switches with QDR, FDR, EDR and HDR speed


=== Architecture ===
=== Architecture ===

Revision as of 13:16, 13 January 2020

Hardware

  • Pre- & Postprocessing node (smp node)
    • 8x Intel Xeon X7542, 48 cores total @ 2.67GHz
    • 1TB memory
    • shared access
  • CascadeLake 40 cores compute nodes (clx)
    • 96 nodes (clx-25, clx384gb40c)
    • 8 nodes (clx-21, clx384gb40c-ai)
      • 2x Intel Xeon Gold 6230, 40 cores total @ 2.10GHz
      • 384GB memory
      • 1.8TB NVMe mounted at /localscratch
  • CascadeLake 36 cores compute nodes (clx-ai) for artificial intelligence and big data applications
    • 8 nodes (clx768gb36c-ai)
      • 2x Intel Xeon Gold 6240, 36 cores total @ 2.60GHz
      • 768GB memory
      • 8x Nvidia Tesla V100 SXM2 32GB
      • 7.3TB NVMe mounted at /localscratch
      • 220GB SSD mounted at /tmp
  • Haswell 20 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2660v3, 20 cores total @ 2.60GHz
    • 84 nodes (hsw128gb20c)
      • 128GB RAM
    • 4 nodes (hsw256gb20c)
      • 256GB RAM
  • Haswell 24 Cores compute nodes (hsw)
    • 2x Intel Xeon E5-2668v3, 24 cores total @ 2.50GHz
    • 152 nodes (hsw128gb24c)
      • 128GB memory
    • 16 nodes (hsw256gb24c)
      • 256GB memory
  • Skylake 40 Cores compute nodes (skl)
    • 100 nodes (skl192gb40c)
  • Visualisation node with AMD graphics (visamd)
    • 6 nodes
      • 2x Intel Xeon Silver 4112, 8 codes total @ 2.60GHz
      • 96GB memory
      • AMD Radeon Pro WX8200
  • Visualisation node with NVIDIA graphics (visnv)
    • 1 node
      • 2x Intel Xeon Silver 4112, 8 cores total @ 2.60GHz
      • 96GB memory
      • Nvidia Quadro RTX 4000
  • Visualisation/GPGPU graphic nodes (visp100)
    • 10 nodes
      • 2x Intel Xeon E5-2667v4, 16 cores total @ 3.20GHz
      • 256GB memory
      • Nvidia Tesla P100 12GB
      • 3.7TB SSD mounted at /localscratch
      • 400GB SSD mounted at /tmp


  • Interconnect: InfiniBand
    • Various generations of Infiniband switches with QDR, FDR, EDR and HDR speed

Architecture

The NEC Cluster platform (vulcan) consists of several frontend nodes for interactive access (for access details see Access) and several compute nodes of different types for execution of parallel programs. Some parts of the compute nodes comes from the old NEC Cluster laki.


Compute node types installed:

  • Sandybridge, Haswell, Skylake
  • different Memory nodes (32GB, 64GB, 128GB, 256GB, 384GB)
  • Pre-Postprocessing node with very large memory (1TB)
  • Visualisation/GPU nodes with Nvidia Quadro FX5800 or Nvidia Tesla P100


Features

  • Operating System: Centos 7
  • Batchsystem: PBSPro
  • node-node interconnect: Infiniband + GigE
  • Global Disk 500 TB (lustre) for vulcan + 500TB (lustre) for vulcan2
  • Many Software Packages for Development
Retrieved from "https://kb.hlrs.de/platforms/index.php?title=NEC_Cluster_Hardware_and_Architecture_(vulcan)&oldid=5436"