The core finding empirically identifies a data-efficiency frontier: for domain-proximal scenes with>= 200 samples, Swin-Graft surpasses YOLOv11x and CNNs retain advantage for domain-distant scenes.
Abstract
Vision Transformers (ViTs) are widely believed to require more labeled data than CNNs for industrial dense prediction. Through controlled experiments on four industrial datasets, we show that the data-efficiency gap stems from pretraining incoherence, which refers to the statistical mismatch between ImageNet-pretrained ViT backbones and COCO-pretrained CNN necks, rather than from inherent self-attention deficits. We characterize the cross-architecture feature gap and propose a lightweight AlignBlock family for pyramid-level feature recalibration. Our core finding empirically identifies a data-efficiency frontier: for domain-proximal scenes with>= 200 samples, Swin-Graft surpasses YOLOv11x (terminal 703-shot: 0.973 vs 0.956 mAP@50); for domain-distant scenes, CNNs retain advantage (hook 141-shot: 0.900 vs 0.600 mAP@50). Grafted neck weights yield up to 2.5x the mAP of a randomly initialized neck.
SnapViT: single-shot network approximation for pruned Vision Transformers is introduced, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retrainin...
Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort et al.· 0 citations
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s saf...
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations
This work proposes EHV, an Efficient Hierarchical ViT architecture designed for massive scaling, and introduces a Sparse Mixture-of-Experts (SMoE) variant, pushing the scale to an industry-leading 30B parameters.
Shuguang Dou, Dongqian Li, Haoyang Jiang et al.· 0 citations
A systematic evaluation on how static PTQ affects the interpretability / explainability of five widely used CNN architectures shows that architecture selection is as important as the quantization strategy, and shows that classification accuracy is not a reliable indicator of interpretability stability under reduced pre...
Kazi Kamruzzaman Rabbi, Md. Zami Al Zunaed Farabe, Mohammad Sohel Rahman· arXiv.org· 0 citations
Vision Transformers (ViT) capture global image context through self-attention but are data-hungry, typically underperforming Convolutional Neural Networks (CNNs) on the small datasets common in fine-grained tasks such as batik motif recognition. This study investigates whether a ViT, trained via knowledge distillation...
Rafi Alifa Bagja, B. Purnama· International Conference on...· 0 citations
DISTAL is a dual-prior framework for structure-agnostic materials property prediction that combines self-supervised compositional pretraining with structure-aware knowledge distillation and achieves the strongest overall performance among all evaluated feature combinations.
Wei-Ran Wang, Xin-Tong Huo, Yue-Ying Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
