Class‐Imbalance‐Aware Adaptive Dataset Distillation for Scalable Pretrained Model in Credit Scoring

Abstract

The advent of artificial intelligence has significantly enhanced credit‐scoring technologies. Despite the remarkable efficacy of advanced deep learning models, mainstream adoption continues to favor tree‐structured models due to their robust predictive performance on tabular data. Although pretrained models have seen considerable development, the use of such models for tabular‐structured credit‐scoring datasets remains largely unexplored. Tabular‐oriented large models, such as TabPFN, have made the application of large models in credit‐scoring feasible, albeit they can only process limited sample sizes. This paper proposes a novel framework that combines a tabular‐tailored dataset distillation technique with a pretrained model, thereby improving the scalability of TabPFN. Furthermore, although class imbalance is a common characteristic of financial datasets, its influence during dataset distillation has not been systematically explored; we therefore integrate imbalance‐aware objectives into the distillation process, resulting in improved performance on imbalanced credit data. Experiments on six publicly available credit‐scoring datasets show that the proposed imbalance‐aware distillation improves AUC by up to 8.7 percentage points over the standard MSE‐based distillation baseline. At the same time, distilled datasets retain roughly 76%–95% of the full‐data AUC while often using under 10% of the original training samples (and at most 31.3%) and consistently outperforming random subsets. Taken together, these results demonstrate that imbalance‐aware dataset distillation provides a compact yet accurate solution for imbalanced credit scoring and enhances the practical scalability of tabular pretrained models such as TabPFN.