Application of Transformer Models for Debt Repayment Prediction

Abstract

Modern financial institutions require reliable tools for forecasting borrowers' repayment capacity, as this directly affects credit risk and the stability of financial systems. Traditional machine learning methods often lose effectiveness when dealing with large volumes of sequential and heterogeneous data. This study proposes the application of a transformer architecture to predict debt repayment based on the history of financial transactions and customer characteristics. The novelty of this work lies in applying the transformer architecture to debt repayment prediction - a task rarely addressed with such models - by representing customer financial history as sequences and employing a dropout mechanism to improve training performance and prediction precision. The model is optimized for the specific properties of financial data and evaluated on a dataset of real payment records. The effectiveness of the proposed approach is assessed using the precision, recall, and F1-score metrics. The experimental results demonstrate the superiority of the transformer model over classical algorithms (C4.5 decision tree, logistic regression, naïve Bayes classifier, multilayer perceptron, and k-nearest neighbors) in terms of prediction accuracy. The obtained findings indicate the feasibility of employing transformer-based models in credit scoring and risk management systems.