NoduViT: An Attention-Driven DeepLab–CustomViT Hybrid Model for Segmentation-Informed Multiclass Pulmonary Patch Classification

Abstract

Lung cancer detection using CT imaging requires accurate lung segmentation and nodule classification. This study evaluates SegFormer configurations and hybrid CNN–Transformer models using the LIDC-IDRI dataset. Initial experiments showed that pretrained and from-scratch SegFormer models failed to detect target classes, while fine-tuning achieved IoU scores of 0.6628 and 0.6244 for the right and left lungs, respectively, with corresponding F1-scores of 0.7972 and 0.7688. However, nodule segmentation remained challenging (IoU 0), with limited improvement using a Nodule Loader (IoU 0.1487, F1 0.2589). Compared with recent models reporting Dice scores above 94%, SegFormer demonstrated lower segmentation performance. For classification, standalone ViT, ResNet, and DeepLab achieved accuracies of 72%, 61%, and 82%, respectively, whereas hybrid ResNet+ViT and DeepLab+ViT achieved 98% and 99.54%. Despite these improvements, class imbalance reduced minority-class nodule recall and weighted F1-scores. Overall, fine-tuned SegFormer effectively segments major lung structures but struggles with nodules, while hybrid CNN–Transformer classifiers achieve high accuracy yet require improved class balancing for reliable clinical generalization.