QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
This work proposes Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization, and shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.