AI Alignment Without Representation: From Scalar Rewards to Social Deliberation

Abstract

Despite the widespread adoption of Reinforcement Learning from Human Feedback, state-of-the-art AI systems remain prone to two persistent failures. The first is hallucination, or the production of fluent but false content. The second is moral drift, or the convergence toward exploitative or harmful equilibria under sustained deployment. We argue that these are not distinct pathologies but may arise from a single structural cause that we term feedback collapse. This occurs when complex plural human values are compressed into static scalar rewards by geographically concentrated annotator labor and frozen offline, producing a democratic deficit in which affected communities are excluded from the normative settlement while the training signal decouples from the latent objectives of truth and rightness. We propose Social Reinforcement Learning (Social RL) as a framework for restoring feedback integrity. By situating agents in environments driven by peer critique, reputation, observation, and sanction, Social RL treats alignment as an ongoing social negotiation rather than a static specification problem. Peer agents with asymmetric tool access supply truth-tracking epistemic signals and harm-reducing ethical sanctions that can re-align training gradients with true objectives. We formalize feedback integrity as a gradient alignment condition, analyze how social feedback restores it when collapsed, outline concrete LLM-scale experimental protocols, and critically examine political risks including majority capture, opponent shaping, and social washing. We derive governance recommendations for researchers, developers, and policymakers seeking to embed socially grounded alignment in democratically accountable institutions.