Structure-Preserving Reinforcement Learning for Language Models: Leveraging Geometry

Abstract

We propose a geometric framework for structure-preserving reinforcement learning in the fine-tuning of large language models (LLMs), grounded in Lie algebraic geometry. The approach systematically identifies geometric constraints in representation spaces by analyzing semantic transformations and detecting cases where embed- dings exhibit approximate Lie group structures. When such structure is present, we introduce reinforcement learning algorithms that respect these constraints through projection-based policy gradient updates. We provide theoretical guarantees of convergence and derive sample complexity bounds that depend on the intrinsic geomet- ric dimensionality rather than the full parameter space. The framework is particularly effective when language representations reveal coherent geometric patterns that encode semantic relations. Although empirical validation remains ongoing, this work establishes a principled foundation for integrating geometric priors into language model alignment, offering improved efficiency, stability, and interpretability from the perspectives of differential geometry and representation theory.