CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replac...
Rabimba Karanjai, Qun-Hui Gu, Hemanth Hegadehalli Madhavarao et al.
· 0 citations