Skin Disease Classification on the Body Area Using a Combination of Convolutional Neural Network and Vision Transformer

Abstract

Skin diseases affecting the body area represent a significant dermatological challenge due to the high visual similarity between conditions, which complicates accurate diagnosis. Prior studies have predominantly relied on pure CNN architectures, which are inherently limited in capturing long-range contextual relationships between distant lesion regions, and no existing work has integrated CNN with Vision Transformer specifically for body area skin disease classification. This study addresses this gap by applying the SEMMA (Sample, Explore, Modify, Model, Assess) methodology to develop a novel hybrid classification model integrating Convolutional Neural Network (CNN) and Vision Transformer (ViT) for classifying four types of skin diseases on the body area, namely Acne and Rosacea, Eczema, Psoriasis, and Tinea Ringworm and Fungal Infections, using the DermNet dataset. Two CNN backbones were evaluated: DenseNet201 and EfficientNetB4, each combined with four Pre-LayerNorm Transformer Blocks that feature learnable positional encoding to capture long-range spatial dependencies among feature tokens. A two-phase training strategy was implemented, consisting of feature extraction followed by fine-tuning, with balanced class weight applied to address class imbalance detected during exploratory data analysis. The original training data was split into training and validation sets at a 90:10 ratio, while the test data was sourced from the built-in test folder of the DermNet repository, yielding 4,301 training images, 479 validation images, and 1,298 test images. Evaluation on fully isolated test data showed that DenseNet201+ViT achieved 78% accuracy and EfficientNetB4+ViT achieved 77% accuracy. The application of the probability-averaging ensemble strategy further improved performance to 82% accuracy and a Macro F1-Score of 82%, surpassing prior CNN-based studies in the same domain. Grad-CAM visualization confirmed that the model focused attention on clinically meaningful lesion areas, indicating that its predictions are grounded in relevant morphological features rather than image artifacts. However, this study is limited by the absence of clinical validation on real patient data, and the current accuracy remains below the threshold required for standalone diagnostic use. The findings demonstrate that hybrid CNN+ViT architectures, combined with ensemble strategies, offer a promising and interpretable approach for automated skin disease classification, warranting further clinical validation before deployment as a diagnostic support tool.