Typed Counterfactual Preference Optimization for Multi-Turn Conversational Recommendation Agents
Abstract
Large language models have enabled conversational recommendation agents (CRAs) that interact with users through preference clarification, tool-based retrieval, and natural-language recommendation. Existing prompt-based and preference-optimization methods can improve the interaction capabilities of CRAs, but they provide limited direct guidance for a critical multi-turn decision: when should the agent ask for more preferences, invoke the retriever, or present recommendations? To address this issue, we propose PAPO, a progress-anchored typed counterfactual preference optimization framework for multi-turn CRAs. PAPO identifies critical dialogue states where the current policy is likely to make strategically suboptimal action choices. For each critical state, PAPO constructs typed counterfactual candidates across Ask, Search, and Respond, and verifies them with action-type-specific criteria: information gain for clarification, item reachability progress for retrieval, and recommendation readiness and response grounding. The verified counterfactual actions are used to build preference pairs for direct preference optimization. Experiments in three AILO-Agent-based interactive recommendation environments show that PAPO improves recommendation-oriented performance over strong prompt-based and training-based baselines while maintaining competitive simulator-assessed satisfaction. Ablation and action timing analyses further confirm the effectiveness of typed counterfactual construction, type-specific verification, and readiness-aware recommendation.