IADD-TR is proposed, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR), which achieves competitive returns with improved sample efficiency on five MuJoCo tasks.
Abstract
Model-based reinforcement learning (MBRL), which learns environment dynamics to generate synthetic experience, is a promising approach to sample-efficient decision making. Numerous methods have been developed to improve dynamics prediction and policy optimization for MBRL through uncertainty estimation, model regularization, and conservative value learning. However, these methods typically treat the transition model and critic as monolithic predictors, overlooking the policy-induced data bias. Consequently, action can become entangled with environmental evolution, while uneven action coverage may distort the counterfactual value estimates used for policy improvement. To address this, we propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR). IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization. Its latent and state-aligned components are identifiable up to an invertible within-block transformation and pointwise, respectively. For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional. TR augments the critic with an action-density-scaled residual correction and optimizes a targeted loss, yielding doubly robust policy-gradient estimation when either the critic or the replay action density is consistently specified. Extensive experiments on five MuJoCo tasks show that IADD-TR achieves competitive returns with improved sample efficiency.
World models improve sample efficiency by training policies on imagined trajectories, but their usefulness depends on learning representations that capture the information needed for future control. We study whether self-supervised joint-embedding prediction (JEPA) can provide this learning signal for multi-agent reinf...
Brandon Kaplowitz, Osaze James Obahor, C. S. D. Witt· arXiv.org· 0 citations
The Dreamer-SAC framework, which integrates a recurrent state-space world model with an off-policy soft actor-critic algorithm trained directly in latent space, uses a combination of real interactions and short-horizon generated trajectories with n-step target estimation and multi-objective supervision.
Jia-Zhuo Li, Lin-Jiang Cao, Qi Liu et al.· 0 citations
Q-Target Pretrained Transformers is proposed, which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective, and learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy.
Yi-Chen Lin, Xu-Yuan Xiong, Xue Wang et al.· arXiv.org· 0 citations
This work introduces Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning and shows that T2SPO consistently improves overall task success over GRPO.
Bo-Wen Zhang, Jun-Wei He, Mao-Qi Liu et al.· 1 citation
This work revisits the conventional offline RL paradigm and proposes decoupling policy improvement from actor training, and trains the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic.
Xu-Yao Lin, Yixiang Shan, Jin-Ru Duan et al.· 2 citations
SCOUT is a dynamics-aware meta-learning framework that enables manipulation policies to rapidly adapt by continuously revising their internal beliefs about environment dynamics via a shared belief latent space and significantly accelerates online adaptation across simulated manipulation benchmarks and achieves robust s...
Yi-Shu Li, Li-Yu Geng, Xin-Yi Mao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.