Tackling big data challenges for next-generation physics experiments

Scientists are developing vital software to exploit the large data sets collected by the next-generation experiments in high energy physics (HEP), predominantly those at the Large Hadron Collider (LHC).

Over the years, the existing code has struggled to meet rising output levels from large-scale experiments. The new and optimised software will have the capability to crunch the masses of data that the LHC at CERN and next-generation neutrino experiments such as DUNE (Deep Underground Neutrino Experiment) and Hyper-Kamiokande will produce this decade.  

The project, known as SWIFT-HEP (SoftWare InFrastructure and Technology for High Energy Physics), involves the School’s Particle Physics Experiment researchers Professors Peter Clarke, Philip Clark, Sinead Farrington, Christos Leonidopoulos and Drs Yanyan Gao, Akanksha Vishwakarma and Ben Wynne. This work is in collaboration with a number of universities across the UK, partners from the Science and Technology Facilities Council’s (STFC) Rutherford Appleton Laboratory, and in co-operation with CERN and other international stakeholders.

As part of a precursor project to SWIFT-HEP, Professor Farrington, Dr Vishwakarma and UK colleagues coordinated a workshop on Efficient Computing in High Energy Physics where the community met to formulate ideas and plans for the future and to engage with industry invitees. Professors Clarke and Farrington are members of the Advisory Board which led to the eventual SWIFT-HEP proposal. 

If scientists used everyday computers to store one exabyte of data they would need almost one million powerful home computers to do it. Without a software upgrade the need of computing resources for the LHC would be expected to grow six times in size in the next decade. This is not only too expensive in hardware costs and software inefficiencies, but would also imply an increased consumption of electricity. The more efficient software the team will develop will help reduce the usage of computing resources and the carbon footprint of data centre across the world. The project is to ensure scientists can exploit the physics capabilities of future HEP experiments, while keeping computing costs affordable.

The project’s key milestone for 2024 is to evolve from proof-of-concept studies to deployment, ready for the start of data-taking at the HL-LHC and next generation neutrino experiments, which are expected to dominate the particle physics scene in the second half of the decade.