AI4Sky: Diffusion-Based Cloud Motion Forecasting Using Multi-Temporal Satellite Imagery

Abstract

Accurate short-term cloud motion forecasting is essential for applications such as weather nowcasting, aviation safety, renewable energy management, and disaster response, yet traditional numerical weather prediction models remain computationally intensive and limited in capturing fine-scale cloud dynamics. This paper presents AI4Sky, a diffusion-based deep learning framework that models cloud prediction as a conditional image generation task, using sequences of past satellite observations to generate future cloud states. Built upon a Stable Diffusion backbone and efficiently fine-tuned with Low-Rank Adaptation (LoRA, r = 32), the framework is trained and evaluated using INSAT-3DR/3DS satellite imagery.AI4Sky is evaluated against established baselines including ConvLSTM and standard Stable Diffusion Models, with Video Diffusion Models discussed conceptually to motivate design choices. Results show that diffusion-based forecasting produces sharper spatial structures, improved temporal coherence, and more physically realistic cloud evolution. Although the use of 256×256 resolution slightly moderates pixel-level metric scores, the model demonstrates superior temporal stability and perceptual realism compared to recurrent approaches.