Selecting and generating web search based features using small language models for tabular data classification

Abstract

In machine learning, the feature space is a key framework that converts data samples into vectors for analysis and modelling. Consequently, the results obtained from machine learning algorithms depend heavily on the ability to select and generate important features. Feature generation as a central technique converts raw data into a refined feature space that facilitates model training and further enhances its structure. Despite progress in automated feature selection and feature generation, major challenges remain, especially in terms of explainability and cross-domain generalization. One alternative is to adapt automatic feature engineering within deep learning, rather than relying on hand-crafted techniques in machine learning. Although deep learning models have proven highly effective in a variety of fields, they are often considered less effective when applied to tabular datasets. To tackle these problems, we present a methodology that uses small language models (SLMs) to select and generate features specifically for tabular datasets in machine learning classification tasks. Experimental results show that we reached highest F1 scores of 0.383 and 0.848 for two well-known tabular datasets for the machine learning multi-class classification task.