Hybrid Graph Neural Network Framework for Anomaly-Based Detection and Visualization of Suspicious Bank Transactions

Abstract

Anomaly detection in banking systems is becoming more challenging because of the nature of interconnectedness of the transaction data, which cannot be accurately captured through traditional means anymore. In this paper, we propose an approach based on graphs and the use of unsupervised machine learning to detect any abnormal transactions. Transactions are modeled as a heterogeneous graph consisting of accounts, devices, IP addresses, and merchants, enabling the extraction of both structural and behavioral features. The proposed framework combines the Isolation Forest algorithm for behavioral anomaly detection with a Graph Neural Network (GNN) autoencoder for structural representation learning. The outputs of both approaches are integrated through a weighted hybrid scoring mechanism to generate a final risk score for each account. Based on percentile-based thresholds, accounts are classified into low, medium, and high-risk categories. Experimental observations indicate that highly risky accounts tend to form connected clusters through shared infrastructure, while low-risk accounts show minimal connectivity. The proposed approach effectively identifies suspicious behavior as well as coordinated fraud patterns within financial transaction networks. Additionally, graph-based visualization enables intuitive analysis and interpretation of suspicious account relationships.