Skip to content
Preprint

Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

Sep 2026 · 0 citations · 11 references
Computer Science

TL;DR

The results show that rigorous control of pretraining and evaluation is essential for interpreting biomedical architecture benchmarks, while distillation may be a more effective route to practical single-cell classification than architecture choice alone.

Abstract

Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicting conclusions about CNNs versus transformers. We present a controlled benchmark on LIVECell phase-contrast microscopy data using source-image-disjoint train/validation/test splits to prevent parent-image leakage and matched optimisation, augmentation, and evaluation protocols across EfficientNet, Vision Transformer (ViT), and EVA-02 models. This allows the effects of architecture, pretraining, fine-tuning, tokenisation, and distillation to be disentangled. We find that the previously reported CNN advantage is largely explained by pretraining rather than architecture: the smallest pretrained model outperforms the strongest model trained from scratch despite far fewer parameters. Pretraining improves macro-F1 by 3-4 points, while the gap between the best pretrained CNN and transformer is below 0.5 points. Architectural choices nevertheless matter: ViT-S/8 outperforms ViT-S/16 and matches the four-times-larger ViT-B/16 at a quarter of the parameters, showing that finer tokenisation benefits small cell crops. Conversely, layer-wise learning-rate decay, central to the EVA-02 fine-tuning recipe, degrades performance, highlighting that transfer heuristics from natural-image recognition may not generalise to microscopy. Finally, knowledge distillation substantially improves the deployment frontier: compact EfficientNet-B0 students distilled from teacher councils outperform every individually trained backbone, including the EfficientNet-B5 and EVA-02 teachers. Overall, our results show that rigorous control of pretraining and evaluation is essential for interpreting biomedical architecture benchmarks, while distillation may be a more effective route to practical single-cell classification than architecture choice alone.

View source

Similar papers

Open access Sep 2026

Dual-Domain Calibration with Source Rehearsal and Knowledge Distillation for Robust Cross-Dataset Classification

Automated peripheral blood cell classification has shown strong performance on curated microscopic image datasets, but model reliability can decrease when images are acquired from external sources with different visual characteristics. This study presents a reproducible deep learning protocol for five-class peripheral...

T. Wable · 0 citations
Oct 2026

Systematic evaluation of deep learning design choices for image-based organ-on-chip quality assessment.

Label-free brightfield microscopy is widely used to monitor organ-on-chip (OOC) cultures, but manual quality assessment can be subjective, labor-intensive, and difficult to standardize across biological conditions. Using a public dataset of 3072 microscopy images from six cell lines, we systematically evaluated four qu...

A. B. Oktay, Arpit Tandon, Brian E. Howard et al. · 0 citations
Open access 2026

DisagreementNet: Lightweight Multi-Teacher Distillation for Retinal OCT Classification—A Leakage-Aware Evaluation

Retinal disease classification from optical coherence tomography usually relies on large pretrained backbones that do not fit the hardware available in most clinics. This paper presents DisagreementNet, a 227,557-parameter Convolutional Neural Network (CNN) trained from scratch, and Disagreement-Aware Knowledge Distill...

Miit Daga, Swarna Priya Ramu · 0 citations
Sep 2026

Exploiting Spatial Structure for Transductive Few-Shot Classification of Whole-Slide Images

SlideTIM is proposed, an adaptation to WSIs of the recent transductive approach LC-TIM, which introduces a combined spatial--latent regularizer together with a prior on the patch class distribution that addresses the complex spatial organization and the strong class imbalance of WSIs.

T. Godelaine, M. Dausort, Karim El Khoury et al. · 0 citations
Preprint Sep 2026

Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound

Self-supervised learning (SSL) can reduce the need for labelled medical images, but the choice of pretext objective remains unclear for lung ultrasound (LUS). Contrastive learning, masked reconstruction, and joint-embedding predictive architectures (JEPA) differ in the space in which their targets are defined, yet exis...

Moein Heidari, J. Rao, Jai Choraria et al. · 0 citations

Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

LeJEPA is adapted to molecular graphs, evaluating GPS and Chemprop-style D-MPNN encoders on the antibiotic-activity dataset and ogbg-molhiv using a multi-seed, bootstrap-based protocol, and pretraining supplies complementary information best realised through feature-level combination, while finetuning gains are weak an...

Michał Kulczykowski, Rafał Łabędzki · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.