Shutter to science: designing data pipelines for the Liverpool Telescope

Abstract

We present the design of an integrated, cloud-based data pipeline for the Liverpool Telescope to replace the legacy on premises service. We give an end-to-end view of how raw images and telemetry flow from the summit to long term archive and back to astronomers, highlighting key design choices, implementation details and practical challenges. The Science Operations Data Centre coordinates containerized services, message-driven workflows and cloud storage to deliver robust data ingestion, automated calibration, provenance tracking and the generation of science-ready data products (SDPs) suitable for immediate use. We focus on the data reduction pipelines: how calibration data are managed, how processing histories are recorded, and how we ensure that outputs remain reproducible as instruments evolve and survey volume increases. The pipelines tolerate intermittent connectivity, preserving data integrity and eventual delivery while still providing useful feedback on system state and observing progress.