Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness
Probes tracking in-weights learning, in-context learning, and copying trace these failures to distinct training dynamics, and show that introducing contrast in target-context similarity both within a context and across batches, restores robustness across the entire spectrum.
Deep K. Malu, Deevyanshu Malu, Aditya Nemiwal et al.
· 0 citations