Skip to content

Direct Preference Density Alignment for Conversational Audio Equalization

Sep 2026 · 0 citations · 34 references
Computer Science

TL;DR

This work proposes Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning, and enables the combination of the online structural grounding of Group Relative Policy Optimization with the targeted offline refinement of DPO.

Abstract

Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.

View source

Similar papers

Jun 2026

Data-Efficient Online Training for Direct Alignment in LLMs

This work proposes DOTA, a data selection framework that minimizes the cost of generating preference data, while still ensuring the quality of training, and proposes a theoretically grounded metric called Preference Perplexity (PFP) that enables it to design a low cost, gradient-based method to effectively estimate the...

Chi Zhang, Jia-Chen T. Wang, Kun He et al. · 0 citations
#artificial intelligence Preprint Sep 2026

A Zeroth-Order Paradigm for LLM Preference Alignment

This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles, and establishes a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective.

Peter Chen, Xi Chen, Wo-Tao Yin et al. · 0 citations
#artificial intelligence Preprint Oct 2026

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric...

Ya-Di Qu, Yu-Sheng Han, Cheng-Jia Feng et al. · 0 citations
#machine learning Preprint Sep 2026

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

This work proposes Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm that outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time, demonstrating both its effectiveness and efficiency.

Wen-Bo Zhang, Wen-Zhuo Zhou, Heng-Rui Cai et al. · 0 citations

Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling

Preference-Aligned Test-Time Training (P-TTT) is proposed, which explicitly encodes preference relations conveyed by contextual pairs into user-specific fast weights for personalized reward prediction and outperforms state-of-the-art methods by a large margin.

Bo-Hao Wang, Xiao-Yan Zhao, Yang Zhang et al. · 0 citations
#machine learning Preprint Sep 2026

FestDPO: Few-step Generator Alignment with Direct Preference Optimization

This work introduces Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples that makes sample-based approximation of DPO loss computationally feasible.

Jaewoo Lee, Kyuil Sim, Hyeongyu Kang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.