Kensuke Sasaki

On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

The concept of dual sparsity is introduced: the DPO update direction is a sparse linear combination of sparse gradient feature vectors of sparse gradient feature vectors arising from gradient cancellation between preferred and dispreferred responses, and a framework relating dual sparsity to weight disentanglement is p...