LCW-DPO: Direct Preference Optimization with Length Calibration and Confidence Weighting
Direct Preference Optimization (DPO) has become an important method for preference alignment of large language models. However, standard DPO still faces two common issues on real-world preference data. First, sequence-level log probability is obtained by accumulating token-level log probabilities, which makes the impli...