R-Select, a robust and scalable framework that optimizes data selection with 30 distinct quality metrics, introduces a novel hierarchical optimization strategy that consistently outperforms both heuristic baselines and model-based methods, offering a robust solution for high-quality data curation.
Xin Gao, Xiao-Yang Wang, Yun Zhu et al.· Proceedings of the 32nd ACM...· 0 citations
In recent years, online Direct Alignment from Preferences (DAP) has emerged as a popular alternative for Reinforcement Learning from Human Feedback (RLHF) due to its training stability and simplicity. In online DAP, training relies on preference data, each composed of a question and a pair of large language model (LLM) responses. However, annotating preference data, i.e., generating responses for questions, and using these data to train the RLHF model are computationally expensive. To address this, we propose DOTA, a data selection framework that minimizes the cost of generating preference data, while still ensuring the quality of training. First, we propose a theoretically grounded metric called Preference Perplexity (PFP) that enables us to design a low cost, gradient-based method to effectively estimate the contribution of each preference data point to model performance — critical to data selection. Second, rather than first generating responses for all candidate questions and then selecting preference data points by measuring their PFP, we design an iterative end-to-end framework that only has to generate responses for a small subset of questions, without missing valuable data points. Experiments on UltraChat-200k and HH-RLHF across 13 downstream tasks demonstrate that DOTA reduces computation cost by a factor of three on LLaMA-3-8B, Qwen-3-4B, and Qwen-3-1.7B, without compromising training effectiveness.
Chi Zhang, Jia-Chen T. Wang, Kun He et al.· Proceedings of the VLDB Endo...· 0 citations
The transition from architecture-centric scaling to data-centric refinement has established high-quality data as a critical determinant of Large Language Model performance, particularly for complex reasoning and instruction following. However, effective data selection remains a persistent bottleneck: simple heuristic filters often fail to capture multifaceted data features (e.g., reasoning depth and diversity), while advanced model-based scoring methods typically prioritize isolated quality dimensions, failing to provide a holistic assessment. To address these challenges, we propose R-Select, a robust and scalable framework that optimizes data selection with 30 distinct quality metrics. Recognizing that optimizing such a high-dimensional feature space is non-trivial, R-Select introduces a novel hierarchical optimization strategy. This approach structurally decomposes the search problem by first clustering correlated metrics into functional groups based on statistical dependencies. It then executes a two-stage optimization process: performing intra-group refinement to maximize local representational power, followed by inter-group integration to balance global quality domains. Crucially, to ensure computational efficiency, R-Select employs a low-resource proxy strategy, utilizing a lightweight model on a small data subset to learn an optimal selection policy that is transferable to the target model. Extensive experiments demonstrate that R-Select consistently outperforms both heuristic baselines and model-based methods, offering a robust solution for high-quality data curation.
Xin Gao, Xiaoyang Wang, Yun Zhu et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
