This work proposes Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning, and enables the combination of the online structural grounding of Group Relative Policy Optimization with the targeted offline refinement of DPO.
Abstract
Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.
This work proposes DOTA, a data selection framework that minimizes the cost of generating preference data, while still ensuring the quality of training, and proposes a theoretically grounded metric called Preference Perplexity (PFP) that enables it to design a low cost, gradient-based method to effectively estimate the...
Chi Zhang, Jia-Chen T. Wang, Kun He et al.· Proceedings of the VLDB Endo...· 0 citations
This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles, and establishes a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective.
Peter Chen, Xi Chen, Wo-Tao Yin et al.· 0 citations
Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric...
Ya-Di Qu, Yu-Sheng Han, Cheng-Jia Feng et al.· 0 citations
This work proposes Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm that outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time, demonstrating both its effectiveness and efficiency.
Wen-Bo Zhang, Wen-Zhuo Zhou, Heng-Rui Cai et al.· 0 citations
Preference-Aligned Test-Time Training (P-TTT) is proposed, which explicitly encodes preference relations conveyed by contextual pairs into user-specific fast weights for personalized reward prediction and outperforms state-of-the-art methods by a large margin.
Bo-Hao Wang, Xiao-Yan Zhao, Yang Zhang et al.· arXiv.org· 0 citations
This work introduces Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples that makes sample-based approximation of DPO loss computationally feasible.
Jaewoo Lee, Kyuil Sim, Hyeongyu Kang et al.· 0 citations