Preference-Gradient Subspaces for Memory-Efficient Zeroth-Order Alignment of Large Language Models

Abstract

Zeroth-order (ZO) optimization aligns large language models (LLMs) with human preferences using only forward passes, keeping memory close to inference cost, but its gradient-estimation variance grows with the parameter dimension D. We propose deriving the ZO perturbation subspace from the preference data itself: a one-time, offline backpropagation over a small set of preference pairs yields per-pair gradients whose top principal directions span a k-dimensional subspace (k ≤128) in which on-device, forward-only optimization proceeds. We introduce the capture ratio ρ as a measurable quality index of a subspace, prove that the expected per-step progress of subspace ZO-SGD scales as ρ/(k+2) with a loss floor governed by 1 – ρ, and show that random subspaces (ρ ≈ k/D) provide no advantage over full-space ZO. A CountSketch–Gram estimator measures ρ at the 1.31 × 109-parameter scale without materializing the subspace. On Qwen2.5-1.5B aligned with direct preference optimization (DPO), preference subspaces attain ρ=4.6 10−2 at k =16—about 106 times the random baseline—and, in×our setting, ρ is invariant when D shrinks by 127×, indicating that it tracks the effective rank of the preference-gradient distribution rather than D. Against six stronger low-dimensional alternatives—SFT-gradient spectra, magnitude/Fisher-style masks, structured low-rank random projections, and calibrated full-space ZO—only preference-gradient PCA both reaches high ρ and turns it into training descent, improving held-out DPO loss from 0.694 to 0.619; the alternatives stall or worsen. Peak training memory is 5.2–8.0 GB, versus 16.3 GB for a strictly matched QLoRA baseline.