Star4D: Real-Time Interactive 4D Scene Generation via Spatiotemporal Autoregressive Modeling
Abstract
Building world models with spatial consistency and real-time interactivity is a frontier challenge in computer vision. Current video generation paradigms often lack spatial persistence and sufficient visual realism, hindering seamless navigation in complex environments. To address these challenges, we propose Star4D, a novel real-time 4D scene generative model that transforms single reference videos into high-fidelity, dynamic, and interactive environments. At the core of our approach is a spatiotemporal autoregressive architecture. This architecture integrates an implicit spatiotemporal cache with explicit spatial guidance, enabling high-fidelity generation and precise camera control while maintaining real-time performance. Furthermore, we introduce Joint Distribution Matching Distillation, which leverages real-world data distributions as regularizing guidance to effectively overcome the fidelity degradation typically caused by over-reliance on synthetic data. Extensive experiments demonstrate Star4D significantly outperforms existing state-of-the-art models in spatial consistency and interaction precision, ranking first among real-time interactive methods on the WorldScore-Dynamic benchmark, and establishing a practical pipeline for navigating 4D environments instantiated from monocular videos.