Multi-Agent Reinforcement Learning via Agent-Specific Preference

Abstract

Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized pReferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical. Note to Practitioners—Multi-agent systems are widely used in modern engineering applications. For example, autonomous vehicle fleets coordinate to prevent collisions while maintaining efficiency, and industrial manufacturing lines work together to meet production targets without causing buffer overflows. Multi-agent reinforcement learning (MARL) provides a powerful framework for enabling such collaboration, but its success depends heavily on well-designed reward functions. Designing these rewards is often challenging, especially when agents play distinct roles, as it is difficult to translate complex interactions and diverse agent behaviors into precise numerical signals. In contrast, providing comparative feedback on preferred behaviors is often more intuitive than specifying explicit mathematical rewards. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), a framework that leverages agent-specific preference signals in the multi-agent learning process. MAGPIE learns agent-specific reward models and combines them into a unified global objective using monotonic aggregation. By optimizing this objective, we can derive Nash equilibrium solutions. Importantly, preferences can be provided by lightweight automated rules or domain-specific heuristics, eliminating the need for costly human annotators. MAGPIE is effective, easy to implement, and particularly suitable for complex systems where traditional reward design is impractical.