GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that m...
Andreea Dutulescu, Stefan Ruseti, Mihai Masala et al.
· 0 citations