Scientists of the European nuclear research organization CERN are preparing the restart of the Large Hadron Collider (LHC) in 2021. As hardware platform they use AMD processors. Among other things, the researchers will use the LHC to carry out the LHCb experiment. It is an attempt to simulate the possible conditions immediately after the Big Bang and to understand why the universe consists mainly of matter, not anti-matter.
In the particle accelerator, collision data is recorded at a rate of 40 TB/s and must be processed by the hardware platform. The task of the hardware is to view all incoming information and filter out the relevant events. All data is stored and processed via FPGAs on servers.
AMD processors keep data throughput constant
The researchers in the LHCb experiment use four Mellanox 200 Gbit InfiniBand adapters per server. Using AMD's second-generation EPYC processors, the scientists achieved a data flow of more than one terabit per second across the servers for days. Achieving this on a single server instead of requiring a supercomputer – as was the case in the past – is a significant advance.
AMD's 64-core EPYC-7742 processors allowed the four Mellanox NICs in each server to run without bottlenecks. They also support up to 128 PCI Express 4.0 lanes.
The entire memory pool is also important: the researchers cannot stream data directly from the FPGA card to the network card. So the data has to go into main memory and come back again. LHCb uses 512 GB of memory on each of its servers. The Rome platform meets the need for this because it has many memory channels and offers support for high bandwidth. With up to 8 TB DDR4 at 3,200 MHz, AMD processors deliver the RAM requirements and match what the PCI Express bus can provide.
More processing power
»We don't have the computing power of Google or Facebook,« says Niko Neufeld, project manager at CERN, classifying the performance. However, the EPYC-CPU allows the necessary data processing in a compact system design. This was not yet possible ten to 15 years ago. »Now there is room for growth. With the EPYC technology it is possible to double our capacity in the same space. For the coming years we plan to increase the detectors and sensors,« Niko Neufeld continues.
By using the CPUs, CERN will reduce the number of servers by a third. This shortens the latency time when the scientists want to build a high-speed network. With a larger network, more collision problems occur - the more compact the system is, the better.