Understanding Helpfulness and Harmless Tension in Reward Models

Abstract

Reward models (RMs) are a key component of reinforcement learning from human feedback, enabling the alignment of large language models with human values such as helpfulness and harmlessness. However, these alignment goals can conflict, leading to interference when optimised jointly. We refer to this conflict as alignment tension and investigate how it manifests in RMs. To this end, we train RMs optimised for helpfulness, harmlessness, and a combined helpful-harmless objective, and compare their behaviours and internal representations. We find that mixed-objective RMs frequently underperform their single-objective counterparts, suggesting substantial interference between alignment goals. Using activation-based interpretability methods, we identify neurons associated with helpfulness and harmlessness and show that they causally promote their corresponding behaviours while often suppressing the opposing one. At the same time, we find that a substantial fraction of neurons are shared across objectives. These shared neurons exert a disproportionate influence on model behaviour and encode incompatible representations. Our findings provide mechanistic evidence for how alignment objectives are represented and interact within RMs, offering new insights into the challenges of multi-objective alignment and motivating the development of more disentangled and controllable alignment methods.