Big Data and advances in Artificial Technology have outstripped current processor technology. Compute Accelerators and GPUs have filled the gap providing thousands of compute cores, field programmable gate arrays, specialised AI-specific cores and high bandwidth interconnection.
Most systems are configured with dual processors, a lot of RAM (starts at 256GB but much more is common), high speed SAS SSDs or NVMe, multiple high wattage power supplies and multiple compute accelerators or GPUs.
A specialised bus is also quite common at the high-end allowing faster interconnect between GPUs.
What Makes a High-Performance Compute server Different?
High Performance Compute servers generally use the highest performance processors available. These generally have a high clock rate and a large number of cores.
AMD currently have the performance edge with up to 64 cores at 3.4GHZ. They also offer a range of 32 core processors.
Intel has the 28 core 8280 at 2.7ghz shipping and announced 56 core processors having discontinued its Phi 64-72 core processors.
HPC Processors are low volume and very expensive items with each processor costing tens of thousands.
We have selected the fastest and latest processors in our high-end systems and offer slightly slower processors in our mid-range systems.
We have a number of our HPC Servers offering Dual AMD 64 core. We also offer multiple Intel 28 core processors in a number of our systems
Most of the high-performance servers run DDR4-2999GHZ RAM although faster RAM is available it is often not supported by the Equipment manufacturers.
Most high-performance Compute servers have a minimum of 512GB RAM although systems with 3TB or more are available.
There is also use of Intel Optane DC Persistent RAM available in 512GB modules to massively expand the maximum memory size and provide non-volatile storage for in-RAM databases.
We have the complete Intel Optane Range available and able to be included in many of our HPC Compute configurations.
The industry standard PCIE3 bus is currently being updated to PCIE4 effectively doubling the transfer rate from 32GB/sec to 64GB/sec.
For systems with a large number of GPUs Nvidia has SXM3/NVLINK bypassing the PCIe bus and providing up to 300GB/sec between GPUs.
We have the largest range of SXM3 systems available in Australia and new PCIE4-based systems
The HPSS Collaboration is an organisation specifically addressing High Performance Computing storage requirements.
PetaFlops understands storage performance is critical and so SAS SSD technology is preferred. Many systems are also utilising M2.NVMe for its advantages in transfer rates.
As you can see from the chart, Intel Optane offer extremely low drive latency and the use of the memory slots avoids significant controller latency.
The use of Intel Optane DC RAM (see above) is perfect for certain HPC operations but M2.NVME and SAS SSDs are more commonly used.
We offer M2.NVMe in several of our High-Performance Systems. Others have arrays of high performance SAS SSD.
The Infiniband 200GB/sec is still popular for multi compute-node supercomputers but 100GB/Sec ethernet products are also considered.
For industry applications a minimum of 10GBE ethernet is recommended.
All Petaflops systems come with a minimum of 10GBE network connections.
Nvidia has recently upgraded its V100 to V100s and this has been measured at 8.2TFLOP for FP64 double-precision floating point operations.
Its V100 and V100S both have 5120 CUDA compute cores and 640 AI specific Tensor cores per board. With 8 board systems performance of over 50TFLOPS with more than 40000 compute cores and 5000 tensor cores are possible.
AMD have the Radeon Instinct Series Accelerators and the latest M150 accelerators support PCIe4 and have been benchmarked at 6.6TFLOPs for FP64 double-precision floating point operations.
We support the AMD accelerators in some of our Gigabyte and HPe server offerings.
Intel’s FPGA (field programmable gate array) cards are often used to implement algorithms in hardware providing the highest possible performance for these tasks. Sometimes this has applicability in Deep Learning models or in financial systems. They require significant programming effort.
Intel FGPA cards are supported in a few of our HPC systems. Please contact us for details.
The Alveo U250 is an impressive PCIe3 GPU and is supported in a limited number of HPC servers including the HPe DL380G10
How Many GPUs?
In most traditional PCIe servers the limit is 4 double width GPUs (or 8 lower powered single width GPUs) supported by specialised power supplies (eg dual 2000W with 15A power required).
Up to 8 SXM GPU modules are supported in a number of supercomputers we have available.
The Nvidia DGX-2 system supports up to 16 GPUs in one system making it the world’s most powerful deep learning server. Nvidia have linked 96 of the DGX-2 servers into one supercomputer (in the top 100) with 8.8million CUDA and Tensor cores and 12000 TFLOP performance.
Systems with 2000W or more power supplies are common and necessary to support the power requirements of the GPUs.
High performance fans are standard and many vendors offer additional cooling options such as liquid cooling.
The systems generally require multiple 15A connections.
Single High-Performance compute can provide more performance than 100 CPU-only nodes.
However, to scale to high-end supercomputers leveraging massive parallel processing it is necessary to interconnect these computers. This is generally achieved with extremely high-speed networking.
Up to 200Gbit/sec networking is available with Mellanox Infiniband and Ethernet. Standards are being finalised for much faster solutions up to 1TBit/sec.