2025· Neural Information Processing Systems· pp. 6994-7020· 0 citations· 37 references
Computer Science
TL;DR
DPAIL, an adversarial IL framework that leverages diffusion models as a policy class to enhance expressiveness, trains a diffusion policy using a binary cross-entropy objective to distinguish expert trajectories from generated ones to enable optimization of the diffusion policy.
Abstract
Human experts employ diverse strategies to complete a task, producing to multi-modal demonstration data. Although traditional Adversarial Imitation Learning (AIL) methods have achieved notable success, they often collapse theses multi-modal behaviors into a single strategy, failing to replicate expert behaviors. To overcome this limitation, we propose DPAIL , an adversarial IL framework that leverages diffusion models as a policy class to enhance expressiveness. Building on the Adversarial Soft Advantage Fitting (ASAF) framework, which removes the need for policy optimization steps, DPAIL trains a diffusion policy using a binary cross-entropy objective to distinguish expert trajectories from generated ones. To enable optimization of the diffusion policy, we introduce a novel, tractable lower bound on the policy’s likelihood. Through comprehensive quantitative and qualitative evaluations against various baselines, we demonstrate that our method not only captures diverse behaviors but also remains robust as the number of behavior modes increases.
This work proposes MAFIS, a novel method that addresses limitations for both online and offline MAIL settings by building upon the single-agent IQ-Learn framework and introducing the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems.
Yichen Li, Zhongxiang Ling, Tao Jiang et al.· Neural Information Processin...· 3 citations
Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: https://github.com/AmineAndam04/advrl
Amine Andam, Jamal Bentahar, M. Hedabou· 0 citations
Agent behavior consists of two modalities, state-action trajectories and paired videos. Nevertheless, the absence of viable reward functions in practical workflows often undermines reliable assessment and robust control, a challenge that imitation learning addresses by learning policies directly from expert demonstrations. However, imitation learning heavily relies on expert demonstrations, which are typically characterized by skewed trajectory distributions, leading to mode collapse and insufficient coverage of infrequent behaviors. To address the challenges, we propose a two-stage framework for multi-modal policy acquisition under trajectory distribution skew. Specifically, the first stage performs self-supervised pre-training for future state prediction to capture dynamic features more effectively, providing a strong foundation for subsequent policy learning. In the second stage, we encourage tight clustering of samples from similar modes in the latent space while separating those from different modes with a mode-aware contrastive objective, thus improving multi-modal disentanglement. We further introduce a frequency-aware dynamic scaling factor that reweights mode-conditioned learning signals according to mini-batch mode frequencies, thereby increasing the contribution of underrepresented modes. Beyond state-based metrics, we additionally report a complementary post-hoc assessment using Video-Mode Prototype Consistency Accuracy (VMPCA), which measures whether a generated rollout video is assigned to the expert-video prototype corresponding to its predefined behavior mode. The experimental results demonstrate that our method captured both common and rare expert behaviors from skewed trajectory distributions across diverse multi-modal scenarios. It achieves competitive or superior VMPCA performance, providing complementary evidence that generated rollout videos are frequently assigned to expert-mode prototypes corresponding to their conditioning modes.
In order to make Reinforcement Learning algorithms applicable in real world scenarios, safety must be ensured even under adverse operating conditions. In this work, we consider the challenge of adversarial feature missingness: a scenario in which an adversary occludes features from the agent's observation in order to reduce performance as much as possible. We formally define adversarial missingness for Reinforcement Learning and compare it to the related concepts of $\ell_\infty$-norm bounded adversarial perturbations and learning with missing data. We develop an adversarial training algorithm and show its effectiveness in increasing robustness against adversarial missingness on three MuJoCo benchmark environments. Compared to a baseline trained with random uniform missingness, our method achieves better robustness on all three tasks.
Paul Stahlhofen, Luca Hermes, Tim Kochs et al.· 0 citations
RL-FAT is proposed, a reinforcement-learning-inspired fair adversarial training framework that uses policy-gradient based feedback from adversarial predictions to improve adversarial robustness while promoting a more balanced robustness distribution across classes.
Diffusion-based robot policies have become widely used in robotic manipulation, where they are typically trained with behavior cloning. However, policies trained purely from demonstrations are limited by the quality and coverage of the available data. Reinforcement learning can further improve the performance of these pretrained policies through interaction. A common approach is to use policy-gradient methods that formulate diffusion-policy fine-tuning as an outer environment MDP together with an inner denoising MDP. However, existing methods typically assign the same environment-level credit to all denoising steps used to construct an action chunk, without distinguishing which intermediate decisions contributed most to the final return. We introduce Denoising Intermediate Advantage (DIA), a policy-gradient method that learns a value function over partially denoised actions and uses it to construct a denoising level advantage for each step of the generative process. DIA combines this inner credit signal with the standard environment-level PPO advantage, providing state-dependent credit throughout the denoising chain. Across Robomimic, FurnitureBench, Franka Kitchen, and D3IL, DIA consistently improves final performance over existing diffusion-policy fine-tuning methods. Beyond final reward, DIA reaches successful states more efficiently and can shift farther from the pretrained behavior distribution, enabling it to discover more effective and efficient task-level strategies and subtask sequences that baseline methods fail to reach.
Arjun Sohal, Yuchi Zhao, Miroslav Bogdanovic et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
