The LHCb experiment is undergoing a significant upgrade during LS2 in order to allow it to run at a five times higher instantaneous luminosity in LHC Run 3. In order to deal with the corresponding increase in pileup per bunch crossing, most of the LHCb subdetectors are being upgraded to improve their granularity, including a complete replacement of the tracking system, the installation of a new pixel vertex detector and a complete overhaul of the fron-end electronics to allow to read out data at 40 MHz rate. The upgrade of the LHCb trigger is one of the greatest data challenges attempted in HEP. At the heart of this upgrade lie the LHCb’s DAQ and trigger system, which will allow the entire detector to be read out at 40 MHz and reconstructed in real time in software using commodity processors for the first time. Reading out and processing 40 Tb/s in real time, the upgraded LHCb DAQ has a comparable readout rate as the full ATLAS and CMS HL-LHC detectors. Although LHCb’s software was incrementally optimized throughout Run 1 and 2, it was clear that such an incremental approach would not scale towards Run 3. For this reason the collaboration has since 2015 engaged in a full rewrite of its reconstruction and selection software to enable it to take advantage of modern highly parallel processing architectures.

A real-time detector reconstruction is necessary because LHCb is optimized to study, with high precision, relatively light but abundantly produced particles like beauty and charm hadrons, whose signal rates in the LHCb detector will exceed 1 MHz in the upgrade/next run(s) of the LHC. This abundance of signal has two consequences. First of all, because the signal particles are light, it is not enough to look at calorimeter or muon information in the first trigger level. Instead, LHCb must reconstruct tracks and combine momentum and decay vertex information to select the most interesting 0.5–1 MHz of events for further processing. Secondly, LHCb must then perform a full offline reconstruction of these selected events in the second trigger stage, using a real-time alignment and calibration of the detector. This in turn allows LHCb to select signals with high purity and discard in real time most of the detector information not associated to these signals; a process called “real-time analysis” which was first deployed at the LHCb experiment as a proof-of-concept in Run 2. 

Compared to Run 2, the upgrade software trigger will have to cope with a  five times higher instantaneous luminosity and a thirty times higher read-out rate. It became clear early on in the design process that using the available computing resources in the same fashion as in Run 2 will not scale with the same factors. Modern computing architectures get their processing power from an increased amount of compute cores where several threads process the data in parallel (thread-level parallelism), and an increased amount of instruction-level parallelism where the same instruction operates on multiple data simultaneously (Single Instruction, Multiple Data). It was therefore necessary to redesign LHCb’s software framework and scalar algorithms to efficiently utilise such highly parallel architectures.

A common problem with highly parallel computing is the interference of threads. This can manifest itself in two threads blocking each other as they need the same resources, or in one thread undesirably changing the state of another. To avoid these problems LHCb has implemented a simple but effective model: every algorithm must explicitly define all its inputs and outputs, and the algorithm must not have any “state” or memory of whether it was executed before. The requirement that algorithms are stateless makes sure that running the same algorithm on multiple events in parallel does not lead to interference. While thread-level parallelism is handled by the underlying software framework (“Gaudi”), exploiting instruction-level parallelism is specific to individual algorithms and has to be done manually by the developer to be most effective. Existing algorithms are often not suited to this process of vectorization. Because the optimal approach can depend on the specific computing architecture, it is interesting to perform R&D in parallel for different architectures. Putting these concepts together in the reconstruction software has led to a near-linear scaling with the number of physical cores, shown in Figure 1, which is the theoretical best case. This demonstrates that, unlike the Run 2 code, LHCb’s upgrade software can efficiently exploit modern x86 computing architectures.

Alongside optimizing its x86 software, LHCb has since around 2013 also been exploring the highly parallel architecture of graphics processing units (GPUs). The key concepts of multi-threaded many-core x86 programming apply similarly to the GPU architecture, which enabled cross-platform development. In particular, algorithms are designed by data flow in the two cases and the Structure of Arrays (SoA) data layout is ideal whether for access by x86 vector instructions or by GPUs threads. Furthermore, the parallelization levels within one event were explored concurrently for vectorized x86 and GPU algorithms. For example, the reconstruction of tracks in the vertex detector benefited greatly from an exchange of improvements found in the optimal GPU and CPU implementations []. Compared to the x86 architecture, GPUs have a limited amount of memory available (on the order of tens of GB), however because the raw event size at LHCb is about 100 kB this still allows thousands of events to be processed in parallel and does not limit the system. Similarly, the I/O performance of the 3rd generation PCIe bus (16 GB/s) used by current state-of-the-art GPUs was already sufficient to satisfy LHCb’s Run 3 requirements. The 4th generation cards coming onto the market now provide significant margin on top of what is needed.

The main tasks of the upgrade first-level trigger are charged particle reconstruction in LHCb’s three tracking detectors, muon identification, primary and secondary vertex reconstruction and inclusive selections which reduce the event rate by a factor of between 30 and 60. Because the first level trigger consists of a relatively small number of algorithms that are inherently highly parallelizable, LHCb was able to implement complete and viable first-level triggers on both x86 and GPU architectures. In both cases a few hundred modern CPU servers or consumer GPU cards are sufficient to process the incoming 40 Tb/s data stream and reduce the data rate for subsequent alignment, calibration and processing by the full LHCb reconstruction. Figure 2 shows the throughput for the first trigger stage on various GPU models as a function of their theoretical 32-bit FLOPs performance. Analogously to the earlier CPU plot, this linear behaviour shows an optimal exploitation of the underlying architecture.