Method Choice, Not Biology, Determines In Silico Perturbation Results: A Systematic Evaluation of Eight Methods Across Four Datasets

Abstract

Most in silico perturbation methods for single-cell transcriptomics have been validated only on individual datasets, leaving their reliability and generalizability unknown. Through systematic cross-method, cross-dataset benchmarking of eight methods spanning six mathematical frameworks across four datasets, we find that six of eight methods—including widely used VAE-based and tensor decomposition approaches—fail to produce detectable transcription factor (TF)-to-pathway signals. Only CellOracle and DDIM consistently detected TF-to-glycolysis directional regulation. Cross-pathway analysis in PBMC monocytes revealed biologically coherent TF-pathway associations beyond glycolysis (SPI1→glycolysis 4.4× enrichment, FOS→AP-1 targets 4.4×), with SOX9 serving as a biological specificity control (no pathway enrichment). Method choice alone could reverse biological conclusions: DDIM and scTenifoldKnk rankings were significantly anti-correlated (ρ=-0.811, p=0.027). CRISPRi Perturb-seq validation in K562 cells confirmed TF knockdown suppresses glycolysis gene expression (JUN δ=-1.72, CEBPB δ=-1.59, SPI1 δ=-1.57, FOS δ=-0.70), but CellOracle-predicted perturbation directions did not match experimental directions (40.9% agreement, not different from chance), revealing a fundamental gap between steady-state correlation and causal perturbation. Diagnostic analyses using VAE latent space profiling, correlation distribution comparison, and gene-gene graph analysis identified distinct failure modes in unsuccessful methods: VAE latent space competition (STAT3 signal-to-noise 0.44 vs. SPI1 4.25), correlation noise (TF-glycolysis |r|=0.038 indistinguishable from background |r|=0.047), and graph non-specificity (0.84× enrichment). A controlled ablation experiment showed that adding a GRN prior to DDIM did not improve target recall (delta=0 for all TFs), confirming that performance differences are multi-factorial. These findings establish preliminary guidance for method selection, including cross-pathway validation, direction-aware benchmarking, and minimum data requirements (≥500 cells, ≥1,000 HVGs). Author Summary Computational methods that simulate gene knockout experiments from single-cell RNA sequencing data are increasingly popular, but researchers lack guidance on which method to choose. We systematically tested eight such methods across four different cell types, including macrophages from osteoarthritis and rheumatoid arthritis patients, blood monocytes, and leukemia cells. We found that only two methods—CellOracle and DDIM—reliably detected how transcription factors control metabolism. These two methods also detected biologically coherent signals across multiple pathways, not just metabolism. Worryingly, two different methods applied to the same data could produce opposite conclusions about which genes regulate which pathways. We also observed that detecting a perturbation signal does not guarantee predicting its direction correctly: CRISPR-based experimental validation showed that computational methods captured which genes respond to TF perturbation but not whether they are upregulated or downregulated. Through systematic diagnostic analysis, we identified why unsuccessful methods failed: the key TF signals are too weak relative to background variation for purely data-driven approaches to detect. Based on our results, we recommend CellOracle for initial screening (requiring at least 500 cells and 1,000 highly variable genes), cross-pathway validation for any TF→target inference, and orthogonal experimental validation when perturbation direction matters. Our evaluation framework and practical guidelines help researchers choose perturbation methods appropriate for their specific biological questions.