Classifying CRISPR-Cas9 Off-Target Cleavage Sites from GUIDE-seq Data: A Class-Imbalanced Machine Learning Benchmark
TL;DR
Five machine learning classifiers are benchmarked on a real, published GUIDE-seq off-target dataset and the low absolute precision achievable in this severely imbalanced, small-positive-class setting is reported, as a realistic picture of what off-target classifiers can and cannot yet deliver from sequence alone.
Abstract
Off-target cleavage is a central safety concern for CRISPR-Cas9 genome editing, particularly in therapeutic applications where unintended double-strand breaks carry clinical risk. We benchmarked five machine learning classifiers — logistic regression on mismatch-count summary features, a random forest and a gradient boosting model on one-hot-encoded sgRNA/candidate-site sequence pairs, a one-dimensional convolutional neural network (CNN) over the positional mismatch map, and a gradient-boosting/CNN ensemble — on a real, published GUIDE-seq off-target dataset (Kleinstiver et al., 2016, Nature) comprising 95,829 candidate off-target sites for five sgRNAs, of which only 54 (0.06%) were experimentally validated as true cleavage sites. On a held-out, stratified test split (n = 19,166; 11 true positives), gradient boosting on combined mismatch and sequence features performed best (ROC-AUC = 0.997, PR-AUC = 0.355, best F1 = 0.50), outperforming a random forest on raw sequence encoding alone (PR-AUC = 0.083) and a sequence CNN (PR-AUC = 0.129). Because the positive class is extremely rare, we report precision-recall AUC as the primary metric rather than ROC-AUC, which is inflated by the large negative class. A positional mismatch analysis showed that experimentally validated off-target sites carried substantially fewer mismatches overall than non-cleaved candidate sites (mean 3.6 vs. 5.9 mismatches across the 23-nucleotide target), and were markedly more mismatch-intolerant in the 10-nucleotide PAM-proximal seed region (11.3% vs. 27.4% per-position mismatch rate) and at the PAM itself (6.8% vs. 16.0%), consistent with established seed-region and PAM-sensitivity models of Cas9 target recognition. We report these findings, including the low absolute precision achievable in this severely imbalanced, small-positive-class setting, as a realistic picture of what off-target classifiers can and cannot yet deliver from sequence alone.