Do Pre-Trained Code Models Add Value Beyond Software Metrics in Class-Level Defect Prediction? An Empirical Study of Input Coverage, Long-Code Aggregation, and Cross-Version Generalization

Abstract

Pre-trained code models are increasingly used in software engineering, yet their incremental value beyond traditional software metrics for future-version class-level defect prediction remains unclear. This study evaluates CodeBERT, GraphCodeBERT, and CodeT5 using 16,237 class-version instances from six open-source Java systems and 11 chronological train–validation–test splits. We analyze model-specific input coverage and long-code representations and test whether learned code features add value beyond metric and size controls. Overflow affected 50.7% of instances for CodeBERT and GraphCodeBERT and 39.0% for CodeT5. Defective instances overflowed more often than clean instances, and this association remained after adjustment for size and project-version effects. Long-code strategies yielded small and inconsistent gains, none of which survived Holm correction. The best code-only model achieved a mean Matthews correlation coefficient (MCC) of 0.315, compared with 0.451 for metric-based Random Forest. Fusion produced no robust incremental gain. Multi-seed fine-tuning improved mean MCC for all encoders, but none of the paired gains remained statistically significant after Holm correction, and the best fine-tuned model remained below the metric baselines. These findings indicate that pre-trained code models should be evaluated with explicit input-coverage reporting, chronological validation, strong metric baselines, and incremental-value testing.