In domains such as finance, healthcare, and industry, feature engineering remains the key bottleneck limiting the performance of machine learning models on tabular data. While Automated Feature Engineering (AutoFE) aims to reduce this manual effort, existing approaches still suffer from distinct limitations: data-driven exploration can waste substantial computation on semantically meaningless feature combinations, whereas knowledge-driven approaches using Large Language Models (LLMs) struggle to construct high-order interactions without rich structural context and are typically guided only by coarse global metrics. We propose CoFE (Collaborative Feature Engineering), a two-phase framework that tightly couples search-based exploration with LLM-driven reasoning. In the exploration phase, CoFE leverages an LLM-constructed semantic feature schema to guide a Monte Carlo Tree Search (MCTS), enforcing semantic constraints while encouraging a diverse pool of complex candidate features that provides the missing structural context for LLMs. In the refinement phase, CoFE introduces feature health reports, a diagnostic artifact that supplies the LLM with actionable sample-level and structural feedback for targeted corrections. Experiments on 16 public tabular benchmarks show that CoFE consistently outperforms state-of-the-art data-driven and LLM-based AutoFE methods on the majority of datasets, while offering favorable computational efficiency.
Weihao Jiang, Ziang Nan, Zhihui Shi et al.· Proceedings of the 32nd ACM...· 0 citations
Large Language Models (LLMs) demonstrate strong capabilities in contextual integration and multi-step reasoning, which endow them with the potential to model heterogeneous traffic data. However, the knowledge acquired during LLM pre-training is primarily qualitative and broad, and does not provide the fine-grained, context-specific quantitative dependencies required for accurate traffic prediction. This mismatch between pre-training knowledge and task requirements can reduce the effectiveness of instruction tuning, the standard approach for adapting general-purpose LLMs to downstream tasks. To address this issue, we propose Rules-Injected Instruction Tuning (RuleGPT), a paradigm that injects IF–THEN rules extracted from training data directly into the LLM's instruction sequence. For each sample, relevant rules are retrieved and encoded as learnable rule tokens that are concatenated with the input, providing explicit guidance to the model. By encoding low-order dependencies in this way, RuleGPT allows the LLM to focus on higher-order, context-dependent interactions. Experiments on real-world traffic datasets demonstrate that RuleGPT consistently improves prediction performance over standard instruction tuning and strong forecasting baselines. The code is available at https://github.com/tigerbunny2023/RuleGPT.
Weihao Jiang, Huizhao Wang, Zhihui Hu et al.· Proceedings of the 32nd ACM...· 0 citations