Belief-Based World Models for Partially Observable Reinforcement Learning
Abstract
Embodied agents operating in immersive environments often rely on incomplete egocentric observations, where the current view is insufficient to infer the underlying scene dynamics and make reliable decisions. World models learn compact latent representations to predict future dynamics and rewards, enabling sample efficient policy optimization. However, in partially observable embodied environments, the same observation-action pair may correspond to different hidden states, leading to ambiguous transition and reward dynamics from the agent's perspective. We present BRAVE (Belief Representation for pArtially obserVable Environments), which constructs latent belief representations by integrating the current observation with informative visual cues retrieved from historical memory. Specifically, BRAVE first selects representative key frames from past experience to preserve both relevance and temporal diversity, and then fuses them to construct a more informative belief state for dynamics prediction and downstream decision making. By explicitly modeling belief construction as a retrieval-and-fusion process over egocentric visual memory, BRAVE improves long-horizon prediction and policy learning in partially observable settings. We evaluate BRAVE on diverse embodied decision-making benchmarks, including immersive navigation, open-ended interaction, and multi-agent coordination tasks. Experimental results demonstrate that BRAVE consistently outperforms strong baselines in both sample efficiency and final performance.