This paper develops a multi-agent reinforcement learning-based (MARL) delegation training that enables agents to make sequential delegation decisions while minimizing the total execution cost and introduces two new frameworks for collaboration and delegation in multi-agent systems.
Abstract
AI agents are expected to play an increasingly important role in future decision-making systems. In this paper, we consider collaborative systems composed of heterogeneous multi-agent systems (MAS), where their members have different capabilities and operating costs. We study how agents can delegate tasks to one another so that certain research tasks can be completed effectively under resource-constrained scenarios. We first develop a multi-agent reinforcement learning-based (MARL) delegation training that enables agents to make sequential delegation decisions while minimizing the total execution cost. We then extend this approach to MARL with prescribed delegation topologies. Furthermore, we introduce two new frameworks for collaboration and delegation in multi-agent systems. The first framework proposes that an agent's policy depends not only on the current state of the underlying Markov decision process but also on the interaction history, including previous joint actions. This history-dependent formulation can improve coordination even in fully observable environments, where conventional MARL methods typically restrict policies to depend only on the current state. The second framework proposes a novel, potentially multi-dimensional monetary mechanism to facilitate the collaboration and delegation for MAS.
Action Generation with Topology Awareness (AGTA), a topology-aware sequential decision-making framework in MARL that integrates inter-agent correlation modeling with topology-guided decision-order optimization, and outperforms the state-of-the-art counterparts.
Kun Hu, Shanghua Wen, Wendi Wu et al.· Mathematics· 0 citations
This paper studies policy learning for distributed task allocation in open multi-agent systems, where agents may join and leave in a time-varying fashion, with submodular stage team utilities. At each time, the active agents select actions from local categorical policies such that the feasible joint agent-action pairs form a partition matroid. Standard continuous relaxations of submodular set functions are based on independent Bernoulli sampling, making them inconsistent with agents'policies.To solve this mismatch, we propose the \emph{partition multilinear extension} (PME), a policy-based relaxation whose continuous support matches feasible actions under categorical policies.We prove that the marginal gains of the stage utility provide an unbiased estimator of the gradient of the PME and that maximizing the PME over action distributions is equivalent to maximizing the stage utilities over agent actions, which are critical to devise principled policy gradient.Building on this, we design \emph{SubMAPL}, a centralized-training decentralized-execution KL-mirror policy-learning method that uses local marginal gains as stochastic PME gradients during training. KL-mirror updates preserve categorical feasibility without Euclidean projection.In the case where agents run tabular-softmax policies, we introduce open policy migration and an open-system KL tracking variation to handle agent arrivals and departures. Using dynamic regret analysis, we establish a lower bound on the cumulative utility which accounts for the openness of the environment and for the gap between optimal stage-wise and global utilities. Simulations on multi-agent coverage demonstrate that SubMAPL outperforms policy-gradient and online-learning baselines.
Jing Liu, Luca Ballotta, Yangyang Yang et al.· 0 citations
Although Multi-Agent Reinforcement Learning (MARL) has achieved notable success in cooperative tasks, many existing methods struggle to generalize to heterogeneous scenarios, where diversified specialization is required. In such tasks, a common way is to disable parameter sharing among agents, but this alone fails to induce sufficient behavioral specialization. To address this limitation, sequential updating schemes have recently been proposed, where agents are updated one by one based on the behaviors of their predecessors. However, exploration strategies tailored to the sequential setting remain underexplored, leaving untapped potential in utilizing the information flow between agents. In this work, we propose to leverage the sequential access to predecessor information to explicitly enhance both exploration efficiency and policy heterogeneity. We introduce a novel MARL framework, Multi-Agent Divergence Policy Optimization (MADPO) with Mutual Policy Divergence Maximization (Mutual PDM). The proposed Mutual PDM consists of two key components, the intra-agent PDM and the inter-agent PDM. The former one encourages each agent to explore novel behaviors by deviating from its own previous policies across episodes, while the latter one promotes behavioral heterogeneity between the current agent and its predecessors. To improve the stability and directionality of the divergence signals, we employ the Conditional Cauchy-Schwarz (CCS) divergence for Mutual PDM. Further, we propose a new extension of CCS divergence for measuring policy divergence of more than two agents, the Generalized Conditional Cauchy-Schwarz (GCCS) divergence. Our GCCS divergence can guide agents with more informative incentives, and enhance the global heterogenization with multiple policies. We theoretically prove that the proposed two divergences can offer a stable entropy-guided incentive for exploration and heterogenization. We empirically demonstrate the connection between overall performance and policy heterogeneity in heterogeneous MARL. Extensive experiments on three challenging MARL benchmarks with over 25 heterogeneous scenarios demonstrate that MADPO consistently outperforms existing state-of-the-art sequential up dating methods in terms of both overall performance and agent heterogeneity.
Haowen Dou, Lujuan Dang, Mingfei Lu et al.· IEEE Transactions on Pattern...· 0 citations
Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized pReferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical. Note to Practitioners—Multi-agent systems are widely used in modern engineering applications. For example, autonomous vehicle fleets coordinate to prevent collisions while maintaining efficiency, and industrial manufacturing lines work together to meet production targets without causing buffer overflows. Multi-agent reinforcement learning (MARL) provides a powerful framework for enabling such collaboration, but its success depends heavily on well-designed reward functions. Designing these rewards is often challenging, especially when agents play distinct roles, as it is difficult to translate complex interactions and diverse agent behaviors into precise numerical signals. In contrast, providing comparative feedback on preferred behaviors is often more intuitive than specifying explicit mathematical rewards. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), a framework that leverages agent-specific preference signals in the multi-agent learning process. MAGPIE learns agent-specific reward models and combines them into a unified global objective using monotonic aggregation. By optimizing this objective, we can derive Nash equilibrium solutions. Importantly, preferences can be provided by lightweight automated rules or domain-specific heuristics, eliminating the need for costly human annotators. MAGPIE is effective, easy to implement, and particularly suitable for complex systems where traditional reward design is impractical.
Ni Mu, Yao Luan, Yiqin Yang et al.· IEEE Transactions on Automat...· 0 citations
Many reinforcement learning systems, from fleet management to traffic signal control, must serve an objective that changes dynamically after deployment, and retraining a policy for each new objective is prohibitively expensive. For a single agent, this problem is well understood: successor features with generalized policy improvement, together with their universal extension, recombine a library of learned policies into a policy for any new objective, with a guarantee that the result is never worse than any policy in the library. However, multi-agent transfer has received far less attention, and the common practice of letting each agent recombine its own library independently inherits the recipe but not the guarantee. We prove that this independent composition can produce joint behavior strictly worse than every policy in the library, because recombining teammates changes the environment each agent faces and invalidates the values it relies on, a failure with no single-agent counterpart. We further show that the only unconditionally safe fixed rule is synchronized composition, which moves the whole team to one jointly trained policy but cannot serve objectives that assign different goals to different agents. To attain safety and flexibility at once, we propose MA-USFA, a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates'objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent. Trained once over the distribution of objectives, it is applied at deployment with no per-task adaptation.
Cooperative task scheduling in communication-constrained distributed multi-agent systems is challenging because each agent must make decisions from partial and dynamic observations while satisfying complex practical constraints. Existing heuristics rely on handcrafted bidding rules and repeated consensus, whereas many learning-based methods assume global observations and lack explicit communication-based coordination. To address these limitations, this paper proposes a neural scheduling framework for distributed multi-robot task allocation (MRTA), consisting of a multi-decoder graph attention model (MDGAM) policy model and a critic-free group relative multi-agent policy gradient (GRMAPG) training algorithm. MDGAM uses an extended graph attention mechanism to jointly update node and edge features, and employs multiple decoders to generate task-selection decisions and communication messages. GRMAPG constructs group-relative advantages from equivalent task-planning instances to replace the critic network used in conventional MARL algorithms, thereby reducing training difficulty and improving convergence performance. Experiments under different problem scales and communication ranges show that the proposed method improves task-completion performance over existing heuristic and learning-based methods, while ablation, complexity, and generalization tests further validate the proposed innovations.