Discrepancy-guided parameter suppression for robust fine-tuning and domain generalization

Abstract

Fine-tuning large pre-trained models on specific tasks can improve in-distribution (ID) accuracy but often degrades out-of-distribution (OOD) generalization. Discrepancy-Guided Parameter Suppression (DPS) is a fine-tuning approach that Discrepancy-Guided Parameter Suppression (DPS) is a fine-tuning approach that freezes the top K percent of model parameters based on the magnitude of their changes. By ranking parameters according to a discrepancy score (the magnitude of change during fine-tuning) and suppressing the highest-discrepancy parameters, DPS encourages the model to learn task-relevant features without overfitting to spurious ID-specific patterns. We evaluate DPS on the challenging iWildCam dataset under camera-induced domain shifts, providing the first in-depth assessment of DPS on a real-world distribution shift benchmark. Using a CLIP ViT-B/16 backbone and identical hyperparameters for a cross-entropy baseline and DPS, we observe improvements on OOD metrics: OOD macro-F1 increases from 35.14% to 36.15% and OOD accuracy from 73.92% to 74.95%, roughly a 1% absolute gain on both. We also provide ablation analyses of the discrepancy metric and suppression rate, showing that normalized, channel-wise suppression at high freezing ratios yields the strongest OOD robustness, consistent with prior findings.