According to an article published on Projects, we have now entered the age of big data, but at present we are struggling to cope with the amount we are producing. Laurence Field explains how the CRISP project is helping researchers to make sense of this information overload.

When people refer to big data or a data deluge in science, the terms are entirely appropriate. As scientific detectors and research instruments improve at a fantastic rate, we as a community are producing more and more of it. The Large Hadron Collider at CERN, even before its current upgrade, was producing 200,000 DVDs’ worth of raw data a second, and the Square Kilometre Array radio telescope project is expected to produce 400 gigabytes a second.

The fact is that scientists are generally concerned only with capturing data, with little consideration for its long-term management and its ramifications throughout the supply chain, affecting data recording, storage, transport and accessibility.

With scientific research only as good as the data that informs it, the struggles of IT systems to cope are a concern. CERN, for example, has to throw away most of the data produced by the LHC, as we are simply unable to capture it. If CERN was able to find the Higgs Boson with only a fraction of the data produced by the LHC, imagine what we could achieve if we were able to capture more, or all, of that data.

A lot of the problems stem from the fact that for many years scientists themselves were left to manage the situation which left a lot of information inaccessible to the rest of the community. In recent years, European physicists have pooled resources to address the data deluge – and the opportunities for innovation and new breakthroughs by overcoming it – through the CRISP project (Cluster of Research Infrastructures for Synergies in Physics).

CRISP brings together 11 research facilities, including the ESRF and CERN, collaborating to develop and streamline IT technology. This isn’t simply about finding the most powerful technological solutions. With pressure on funding and science budgets, it’s also about minimising cost, freeing up budget for true discovery, innovation, and research. As the project enters its final year, key issues of concern have been identified and new collaborations and projects are starting to take major steps to address them.

Read more on Projects