The Forager’s Dilemma: Peer Enforcement Alone Is Not Enough for Stable Multi-Agent Alignment

Abstract

AI alignment methods typically rely on one of two strategies, either optimizing a scalar reward proxy learned from human feedback or situating agents in social environments where peer critique provides a dynamic correction signal. We argue that both strategies, taken in isolation, are structurally unstable. Static scalar proxies can decouple the training signal from the latent objectives of factual accuracy and normative safety, producing failure modes analogous to hallucination and moral drift in large language models. Decentralized peer feedback, while adaptive, is vulnerable to the second-order free-rider problem. Because policing is costly, rational agents abstain from enforcement and let norms erode. To study these vulnerabilities in a controlled setting, we introduce the Forager's Dilemma, a multi-agent environment that couples epistemic and ethical failure modes within a shared resource economy. We evaluate four governance regimes across tabular Q-learning and deep RL agents trained with PPO in a Petting Zoo implementation. Our proposed Dual-Enforcement Reinforcement Learning, which combines intrinsic alignment priors with emergent social enforcement, achieved an approximately 1% lying rate, a sustainable gather rate of approximately 76% under Q-learning and 90% under PPO, and the highest cumulative reward across all conditions, with enforcement effort declining progressively as cooperative norms internalize. Our results suggest that stable AI governance may require a synthesis of institutional constraints and social dynamics, and that neither alone appears sufficient in our setting.