Out of Tune: How Different Fine-Tuning Methods Affect Information Security

Abstract

Fine-tuning large language models (LLMs) enables task specialization but introduces privacy risks through training data memorization. Practitioners deploying fine-tuned LLMs in sensitive domains such as medicine, law, and finance must choose among fine-tuning methods without clear guidance on the privacy implications of that choice. We compare full fine-tuning, partial fine-tuning, and Low-Rank Adaptation (LoRA) by measuring how much sensitive personally identifiable information (PII) each method leaks from training data, using planted canaries to track memorization. Six Llama configurations (3B and 8B parameters) are evaluated across the medical, legal, and finance domains under three extraction attack types: prefix attacks, random prompting, and completion attacks. Full fine-tuning leaks the most on average, followed by LoRA. Partial fine-tuning leaks the least in every configuration we tested, though the relative ordering of full fine-tuning and LoRA varies. Contrary to our initial hypothesis, LoRA's parameter efficiency does not yield the strongest privacy protection; we attribute this to LoRA's architectural breadth across all transformer layers. The performance-privacy trade-off these findings expose has direct consequences for deployment. Even partial fine-tuning on the 3B model, our best configuration, leaked canaries, so practitioners should assume some leakage by default.