Reinforcement Learning Fine‐tuning

Abstract

This chapter extends the fine-tuning pipeline by introducing reinforcement learning fine-tuning, which addresses limitations of supervised fine-tuning when optimizing for subjective qualities like helpfulness, safety, or user preference alignment. The chapter presents the complete supervised fine-tuning followed by reinforcement learning fine-tuning (SFT-RL) pipeline used by modern systems. It covers preference-based optimization, reward modeling, policy optimization, and alignment techniques. The chapter emphasizes how RL fine-tuning enables optimization for qualities that cannot be achieved through supervised learning alone, including safety awareness, user preference alignment, and helpfulness. Applications in connected and autonomous vehicle (CAV) systems include aligning model behavior with safety requirements and user preferences in autonomous vehicle interfaces.