The scarcity of labeled data has made semi-supervised learning essential for medical image segmentation. Recently, Vision Transformers (ViT), pre-trained on large-scale 2D natural images have shown remarkable performance in 2D image segmentation tasks. It is natural to transfer the learned knowledge in ViT to the data-limited semi-supervised medical image segmentation task. However, directly applying ViT faces several challenges, including adapting 2D-pretrained models to 3D medical data and addressing performance degradation in ViT when trained on small datasets. To tackle these challenges, this paper proposes a method for medical image segmentation. The method leverages the strengths of both ViT and Convolutional Neural Networks (CNN) via a hybrid architecture. The CNN-based encoder and decoder play a projector and tokenizer role for the ViT, while the architecture of ViT is fully retained to preserve the knowledge in the pre-trained model derived from SAM (Segment Anything Model) as much as possible. In addition, pseudo-labeling serves as the core guidance for our method to learn from unlabeled data. Experimental results show that our approach outperforms the state-of-the-art on the Pancreas-CT dataset by a large margin and enables rapid transfer learning from 2D-pretrained models to 3D medical tasks with few labeled data, making it especially valuable for rare disease diagnosis.
Ruize Shi, Xiaoyan Li, Boyue Wang et al.· International Conference on...· 0 citations
Generalized zero-shot learning (GZSL) addresses the challenging task of recognizing both seen and unseen classes by leveraging shared semantic knowledge. A core challenge in this domain is achieving robust visual-semantic alignment to transfer knowledge from seen classes to novel classes. Current state-of-the-art methods typically fine-tune large-scale visual backbones on scarce training data. However, this approach frequently leads to severe overfitting to seen classes, which significantly degrades performance on novel categories. To mitigate this issue, we propose the Dual Adaptive Visual-Semantic Prompt Collaboration Network (VSPCN+), a novel framework that utilizes prompt-tuning for effective feature adaptation. Our method introduces a dual-prompt mechanism comprising both visual and semantic prompts. The semantic prompts guide the visual encoder to learn visual features that are more semantically consistent with class attributes, while the visual prompts steer the semantic encoder to generate semantic representations that are more visually grounded. This collaborative process enhances the overall visual-semantic consistency. A key innovation of our work is the dynamic generation of instance-adaptive prompts, which contrasts with existing prompt-learning methods that rely on static, global prompts. By tailoring prompts to individual instances, our approach enhances the model’s robustness and generalization capabilities across diverse visual inputs. This collaborative adaptation, guided by our dual-prompt mechanism, allows the visual and semantic encoders to produce consistent representations for effective visual-semantic alignment. Extensive experiments on standard GZSL benchmarks demonstrate that our proposed VSPCN+ performs favorably against several state-of-the-art methods.
Huajie Jiang, Zheng-Xian Li, Yuankai Qi et al.· International Journal of Com...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
