Skip to content
Preprint

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

Oct 2026 · 0 citations · 55 references
Computer Science Mathematics

TL;DR

This paper introduces objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency, and designs routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward.

Abstract

Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.

View source

Similar papers

Review Open access Oct 2026

Reinforcement Learning: A Comprehensive Review of Foundations, Algorithms, and Open Challenges

Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised lear...

Mohammad Hatim Eliwi, Waleed A. Hammod, O. A. Hammood et al. · 0 citations
#artificial intelligence Review Oct 2026

Optimal Transport Meets Reinforcement Learning: A Survey

Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergenc...

Yu-Jie Zhu, Charles A. Hepburn, Matthew Thorpe et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment

TrajEvolve reformulates step-level continuous credit assignment as a binary classification task, whether each step is necessary for task completion, and simplification not only reduces the learning difficulty but also enables step labels to be obtained through counterfactual verification, achieving precise identificati...

Yi Wen, Hao Chen, Wan-Yu Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learnin...

Amr Mousa, Rifny Rachman, Neil Karavis et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.