#artificial intelligence
Apr 2026
Why Fine-Tuning Encourages Hallucinations and How to Fix It
This work proposes a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t.~pre-existing knowledge by regularizing output-distribution drift and investigates the mechanism, contrasting capacity limitations, behavior cloning, and localized interference.
Guy Kaplan, Zorik Gekhman, Zhen Zhu et al.
· arXiv.org · 5 citations