Automating data reduction and archival workflows using Apache NiFi

Abstract

Traditional data reduction pipelines are run on local or remote machines and depend on user interaction for execution. This approach can lead to reproducibility and discoverability issues, as processing decisions made by the user go undocumented and are made in a single instance leading to difficulty in reproduction. Observatory sites face additional challenges as telescopes are often remote and have constrained or unreliable network connections. These difficulties can result in observing teams manually copying data from on-line observing system, or attempting to use tools over latency-prone networks, often diverting significant time from the business of producing scientific results. Data Central’s Apache NiFi system addresses these problems by combining secure site-to-site data transfer, archival workflows, and automated triggering of pipeline execution to produce data products shortly after observations. This system also integrated with the Data Central Archive to perform archival processes, ensuring data can be easily located, shared and reused. We present the implementation of this system for Data Central’s archive of The Australian National University (ANU) 2.3m Telescope at Siding Spring Observatory and discuss possible future directions.