Malicious Repository Detection Using Transformer-Based Models: A Metadata-Driven Approach

Abstract

GitHub and similar platforms host millions of software repositories, a fraction of which openly distribute malicious source code. Existing work on malicious repository detection has primarily relied on handcrafted features, structural analysis, or multimodal code embeddings, leaving the potential of transformer-based natural-language models largely unexplored. We present a metadata-driven detection framework that fine-tunes DistilBERT on repository metadata text, specifically the description, README, and topics, and formulates detection as a binary classification task between malware and benign repositories. Our dataset combines malware repositories sourced from hacker-community crawls with benign repositories from the CrossSim benchmark, yielding 1,478 BERT-ready samples after leakage-aware preprocessing. Training under a 10-fold GroupKFold protocol stratified by repository identifier to reduce same-repository leakage across training and validation splits yields a mean accuracy of 98.98 % and a mean F1-score of 99.22 %, outperforming all Repo2Vec baseline variants by up to 9.22 percentage points in F1-score. A bagging ensemble of three DistilBERT models further improves the F1-score to 99.75 %. Calibration analysis via temperature scaling indicates well-calibrated probabilistic outputs (ECE = 0.0041 for the cross-validation model and ECE = 0.0074 for the ensemble after scaling), suggesting that the framework is promising for threshold-based screening scenarios. Interpretability and robustness analyses further show that the model’s decisions are driven by distributed contextual signals rather than a handful of overt security terms: malware recall remains at 99.18 % even after masking all 22 such keywords, providing direct evidence against a keyword-shortcut explanation of the model’s performance. To the best of our knowledge, no prior work has applied a BERT-family transformer model to supervised repository-level malware-versus-benign binary classification using metadata alone. The main limitation is that the benign benchmark is restricted to Java-language repositories, whereas the malware repositories span multiple languages.