Synergy in data processing pipelines between two observatories

Abstract

LOFAR and the SKA Observatory are two of the most data–intensive scientific facilities ever conceived. They present extreme challenges in terms of both the total amount of computing and the complexity of the algorithms required to analyse the data. The best way to address these challenges is through cooperation. In this paper we aim to foster collaboration in the development of data processing systems, by defining a common technological framework for building and executing pipelines. This is a challenge as the pipelines involve highly complex scientific software, and the huge data volumes pose strong requirements on scalability and parallelisation. The common framework will abstract away the various heterogeneous and distributed underlying compute and storage infrastructures, enabling scientists and software developers from both of the observatories and from across the wider scientific community to work together on common solutions. The paper describes the approach being taken to align data processing software and data infrastructure across the two observatories. We discuss the various technologies that have been adopted or are under consideration, and how two the observatories aim to maximise opportunities for collaboration.