Human Visual Attention Modeling and Content Understanding for 360◦ Environment

Abstract

360° images and videos are increasingly used in virtual reality (VR). Because viewers perceive only a limited viewport at any moment, spatiotemporal viewing behavior critically shapes experienced content and perceived quality. Despite rapid progress in human visual attention modeling, four limitations hinder practical deployment: (1) Lack of accurate modeling of human eye-movement trajectories (scanpaths) in 360° images. (2) Absence of datasets and frameworks capable of individualized scanpath prediction in 360° environments. (3) Limited exploration into the downstream applications of 360° scanpaths. (4) Insufficient research on real-time saliency prediction for 360° videos. To address these limitations, this thesis proposes multiple novel frameworks designed to tackle specific modeling challenges. It establishes a new identity-linked dataset to facilitate future research and explores two downstream applications driven by human eye movements. Extensive experiments, including both qualitative and quantitative results, validate the superior performance of the proposed frameworks, the utility of the datasets, and the efficacy of the applications. First, for 360° image scanpath modeling, this thesis proposes ScanTD, a time-series-diffusion framework that conditionally generates multiple plausible 360° scanpaths. ScanTD utilizes a Vision Transformer (ViT)- based method for image feature extraction, enabling better learning of scene semantic information. Notably, a transformer-based time-series (TTS) module with a novel attention mechanism is integrated into ScanTD to capture the temporal dependency of gaze points effectively. Second, for individualized 360° image scanpath modeling, this thesis introduces the first dataset dedicated to individualized scanpath prediction in 360° imagery, featuring explicit mappings between user identities and scanpaths. Through a comprehensive statistical analysis of this dataset, the thesis reveals the spatiotemporal regularities of human viewing patterns in panoramic scenes. Based on the proposed dataset, the thesis proposes IS360, an individualized 360° scanpath prediction framework that captures stable visual habits and produces personalized predictions. Third, for target-scanpath-guided 360° image enhancement, this thesis develops PSEM, a progressive scanpath-guided enhancement framework. In the first stage, PSEM conducts the source scanpath clustering. Then in the second stage, PSEM embeds the difference between target scanpath and source scanpath into the Segment Anything Model (SAM). Finally, in the third stage, PSEM fine-tunes a Stable Diffusion (SD) Model for 360° image enhancement. Fourth, for scanpath-based blind 360° image quality assessment (IQA), this thesis proposes RL-ScanIQA (reinforcement-learned scanpaths for image quality assessment). RL-ScanIQA jointly optimizes a proximal policy optimization (PPO)-trained scanpath policy and a quality assessor, where the policy receives quality-driven feedback to learn task-relevant viewing strategies. To improve training stability, RL-ScanIQA incorporates multilevel rewards. Moreover, RL-ScanIQA boosts cross dataset robustness using distortion-space augmentation together with rank-consistent losses. Last, extending to videos, this thesis proposes AROS (autoregressive online saliency) for 360° video saliency prediction. AROS maintains recurrent spherical-graph memory in a frame-by-frame pipeline. AROS disentangles saliency prediction into a static branch representing semantic features and a dynamic branch for motion-induced information. To mitigate error accumulation, AROS introduces dual cycle-consistency regularization.