2025· Neural Information Processing Systems· 3 citations· 53 references
Computer Science
TL;DR
This work proposes MAFIS, a novel method that addresses limitations for both online and offline MAIL settings by building upon the single-agent IQ-Learn framework and introducing the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems.
Abstract
Learning from multi-agent expert demonstrations, known as Multi-Agent Imitation Learning (MAIL), provides a promising approach to sequential decision-making. However, existing MAIL methods including Behavior Cloning (BC) and Adversarial Imitation Learning (AIL) face significant challenges: BC suffers from the compounding error issue, while the very nature of adversarial optimization makes AIL prone to instability. In this work, we propose M ulti-A gent imitation by learning and sampling from F actor I zed S oft Q-function (MAFIS), a novel method that addresses these limitations for both online and offline MAIL settings. Built upon the single-agent IQ-Learn framework, MAFIS introduces the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems. Moreover, we observe that the soft Q-function implicitly defines the optimal policy as an energy-based model, from which we can sample actions via stochastic gradient Langevin dynamics. This allows us to estimate the gradient of the factorized optimization objective for continuous control tasks, avoiding the adversarial optimization between the soft Q-function and the policy required by prior work. By doing so, we obtain a tractable and non-adversarial objective for both discrete and continuous multi-agent control. Experiments on common benchmarks including the discrete control tasks StarCraft Multi-Agent Challenge v2 (SMACv2), Gold Miner, and Multi Particle Environments (MPE), as well as the continuous control task Multi-Agent MuJoCo (MaMuJoCo), demonstrate that MAFIS achieves superior performance compared with baselines. Our code is available at https://github.com/LAMDA-RL/MAFIS .
DPAIL, an adversarial IL framework that leverages diffusion models as a policy class to enhance expressiveness, trains a diffusion policy using a binary cross-entropy objective to distinguish expert trajectories from generated ones to enable optimization of the diffusion policy.
Yunseon Choi, Minchan Jeong, Soobin Um et al.· Neural Information Processin...· 0 citations
Agent behavior consists of two modalities, state-action trajectories and paired videos. Nevertheless, the absence of viable reward functions in practical workflows often undermines reliable assessment and robust control, a challenge that imitation learning addresses by learning policies directly from expert demonstrations. However, imitation learning heavily relies on expert demonstrations, which are typically characterized by skewed trajectory distributions, leading to mode collapse and insufficient coverage of infrequent behaviors. To address the challenges, we propose a two-stage framework for multi-modal policy acquisition under trajectory distribution skew. Specifically, the first stage performs self-supervised pre-training for future state prediction to capture dynamic features more effectively, providing a strong foundation for subsequent policy learning. In the second stage, we encourage tight clustering of samples from similar modes in the latent space while separating those from different modes with a mode-aware contrastive objective, thus improving multi-modal disentanglement. We further introduce a frequency-aware dynamic scaling factor that reweights mode-conditioned learning signals according to mini-batch mode frequencies, thereby increasing the contribution of underrepresented modes. Beyond state-based metrics, we additionally report a complementary post-hoc assessment using Video-Mode Prototype Consistency Accuracy (VMPCA), which measures whether a generated rollout video is assigned to the expert-video prototype corresponding to its predefined behavior mode. The experimental results demonstrate that our method captured both common and rare expert behaviors from skewed trajectory distributions across diverse multi-modal scenarios. It achieves competitive or superior VMPCA performance, providing complementary evidence that generated rollout videos are frequently assigned to expert-mode prototypes corresponding to their conditioning modes.
This paper presents a narrative survey of recent developments in MARL and examines research directions centred on centralised training with decentralised execution (CTDE), value decomposition, learned communication, graph-based methods, and model-based learning.
Abdur Rakib, Khoa Phung, Marco Pérez Hernández et al.· Applied Sciences· 0 citations
Deep Reinforcement Learning (DRL) holds significant promise for achieving human-like Autonomous Vehicle (AV) capabilities, but suffers from low sample efficiency and challenges in reward design. Model-Based Reinforcement Learning (MBRL) offers improved sample efficiency and generalizability compared to Model-Free Reinforcement Learning (MFRL) in various multi-agent decision-making scenarios. Nevertheless, MBRL faces critical difficulties in estimating uncertainty during the model learning phase, thereby limiting its scalability and applicability in real-world scenarios. Additionally, most studies on Connected Autonomous Vehicles (CAVs) focus on single-agent decision-making. In contrast, existing multi-agent MBRL solutions lack computationally tractable algorithms with Probably Approximately Correct (PAC) guarantees, a crucial factor for ensuring policy reliability with limited training data. To address these challenges, we propose MA-PMBRL, a novel Multi-Agent Pessimistic Model-Based Reinforcement Learning framework for CAVs, incorporating a max-min optimization approach to enhance robustness and decision-making. To mitigate the inherent subjectivity of uncertainty estimation in MBRL and avoid incurring catastrophic failures in AV, MA-PMBRL employs a pessimistic optimization framework combined with Projected Gradient Descent (PGD) for both model and policy learning. MA-PMBRL also employs general function approximations under partial dataset coverage to enhance learning efficiency and system-level performance. By bounding the suboptimality of the resulting policy under mild theoretical assumptions, we successfully establish PAC guarantees for MA-PMBRL, demonstrating that the proposed framework represents a significant step toward scalable, efficient, and reliable multi-agent decision-making for CAVs.
Ruo-Qi Wen, Rongpeng Li, Xing Xu et al.· IEEE Transactions on Mobile...· 1 citation
A major bottleneck in imitation learning (IL) is the requirement of a large number of expert demonstrations, which can be expensive or even impossible to collect. Learning from a few expert demonstrations in addition to many supplementary potentially suboptimal demonstrations has emerged as a powerful paradigm to address this challenge. However, previous methods often fail to fully utilize the potential of these additional demonstrations by simply discarding nonexpert data. In contrast, our key insight is that suboptimal demonstrations can still be leveraged for policy training if they outperform the current policy. To realize this idea, we propose a novel approach named IL via meta-learning an action ranker (ILMAR), which performs weighted behavior cloning (BC) on a limited set of expert demonstrations along with supplementary demonstrations. It utilizes the functional of the advantage function to selectively integrate knowledge from the supplementary demonstrations. To make more effective use of supplementary demonstrations, we introduce a meta-goal in ILMAR to optimize the functional of the advantage function by explicitly minimizing the Kullback-Leibler (KL) divergence between the current policy and the expert demonstrations. Comprehensive experiments across a wide range of tasks demonstrate that ILMAR significantly outperforms previous methods in handling suboptimal demonstrations. The code is available at https://github.com/F-GOD6/ILMAR.
Jiang-Dong Fan, Guangyi Zhao, Hongcai He et al.· IEEE Transactions on Neural...· 0 citations
M3W is a novel approach that applies mixture-of-experts (MoE) to world model instead of policy, enabling both learning and planning, and demonstrates superior performance, sample efficiency, and multi-task adaptability.
Zi-Jie Zhao, Zhong Zhao, Kaixuan Xu et al.· Neural Information Processin...· 9 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
