Cross-Database Evaluation of Class-Imbalance Mitigation Strategies for Normal/Abnormal Heart-Sound Classification

Abstract

Automated classification of phonocardiogram (PCG) recordings is often evaluated using the PhysioNet/ Computing in Cardiology (CinC) Challenge 2016 dataset. Class imbalance is commonly addressed with approaches such as class-weighted loss or oversampling, but these methods are often evaluated using a single random split of pooled recordings, which can allow recordings from the same source or patient to appear in both the training and test sets. In this study, the six labeled PhysioNet/CinC 2016 training databases, consisting of 3,240 single-channel PCG recordings (2,575 Normal and 665 Abnormal), were used to train ResNet-18 classifiers on 4-second Mel-spectrograms. The three configurations were identical except for their approach to class imbalance: standard cross-entropy (Baseline), inversefrequency class-weighted loss (Class-Weighted), and weighted random oversampling (Oversampled). Each configuration was evaluated under both a pooled random split (within-distribution) and a sourceheld- out (leave-one-database-out) protocol, using three random seeds. Results were summarized with 95% Wilson confidence intervals, and pairwise McNemar tests were interpreted using a Bonferroniadjusted threshold (α = 0.0167). Under the within-distribution split, all three configurations performed well (accuracy 0.912–0.928) and were statistically indistinguishable. Under source-held-out evaluation, performance dropped to or below the majority-class baseline for every held-out source (accuracy 0.19– 0.67), and no method reliably improved cross-source performance. The higher sensitivity observed with oversampling is reported as a preliminary finding. Overall, these exploratory results suggest that the strong within-distribution performance was substantially influenced by source-level information, and potentially subject-level overlap, shared across the split, highlighting the importance of source-held-out or subject-disjoint evaluation when assessing heart-sound classification performance.