Skip to content
Open access

Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction

Oct 2026 · bioRxiv · 0 citations · 25 references
Biology

TL;DR

Geneformer, scBERT, scGPT, and scFoundation are evaluated as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses.

Abstract

Single-cell foundation models (scFMs) are commonly benchmarked using final-layer embeddings, although useful depth may depend on the task. We evaluate Geneformer, scBERT, scGPT, and scFoundation as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses. Intermediate layers significantly improve cell-type accuracy for scGPT and scBERT in several settings, while Geneformer and scFoundation generally show performance comparable to their final layers. For perturbation prediction, LODO identifies layer 1 as an earliest-on-par candidate across all four models, without establishing universal superiority. Paired scGPT CLS and mean-pooling analyses show that readout affects preferred depth. CKA and representation metrics reveal model- and task-dependent geometric changes, while retrieval and collapse checks support the robustness of perturbation findings. We recommend reporting extraction depth, readout, selection protocol, and task-specific diagnostics. A validated non-final layer may also reduce computation, although efficiency was not directly benchmarked.

Read PDF

Similar papers

#machine learning Preprint Nov 2026

Attribution-based diagnostics for single-cell foundation models

Single-cell RNA foundation models (scFMs) are increasingly used for in-silico biological discovery, yet predictive performance alone does not reveal whether their decisions rely on biologically meaningful signals or model-specific shortcuts. We systematically investigate post-hoc attribution of downstream predictions a...

F. Jafari, T. Lazard, Kenza Bouzid et al. · 0 citations
Open access Oct 2026

Single-cell foundation models transfer across deep evolutionary distances

Single-cell foundation models are trained mostly on human and other vertebrate cells, yet are proposed as general-purpose models. Whether they transfer far beyond their training species is unknown. We evaluated zero-shot embeddings from Geneformer, scGPT and UCE against PCA and scVI in developmental atlases of Dictyost...

Lena Trnovec, Mariko Katoh-Kurasawa, Rachel N. Arey et al. · 0 citations
Open access Nov 2026

DECONVersation: Single-Cell Foundation Model-Derived Embeddings for Robust Cell Type Deconvolution of Bulk RNA-seq

Motivation Bulk RNA-seq deconvolution uses single-cell references to estimate cell type proportions in bulk tissue. Existing methods face challenges with batch effects, subject-level variation, and marker gene selection. We propose a new approach leveraging general-purpose single-cell foundation models. Results DECONVe...

Ali Oku, Heather Geiger, N. Robine et al. · 0 citations
Open access Sep 2026

Benchmarking single-cell foundation models for aging biology

Overall, scFMs supported diverse aging analyses, but performance depended on the biological question, highlighting their utility for rare cellular state identification and regulatory analysis.

Xiao-Min Ni, Yun-Hao Liang, Jia-Lin Zhu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.