Visual and lesion descriptor fusion using cross-attention for skin disease diagnosis

Abstract

Introduction Accurate and early diagnosis of skin disease remains a significant challenge in clinical dermatology due to variation in skin tones and visual similarity among disease lesion types. Dermatologists rely on both clinical expertise and advanced diagnostic tools to achieve reliable diagnosis. To assist in the early diagnosis, many researchers have proposed traditional computer vision methods; however, these approaches remain limited due to the wide range of conditions and the overlapping of early-stage symptoms. Recognizing these limitations, deep learning models have been explored to improve performance, but they still face challenges in effective feature extraction and have weak generalization due to limited datasets. Methods To address this, a cross-attention-based framework that integrates visual with structured lesion descriptor derived from dermoscopic attributes is proposed to enhance early-stage skin disease diagnosis. Visual features are extracted using a fine-tuned transformer, while descriptor features are encoded using BERT-based language models. Furthermore, the ISIC 2018 dataset is enriched with structured lesion descriptors automatically generated from dermoscopic images using the ABCD rule. The symptom descriptors provide semantic representations for multimodal framework instead of an independent clinical modality. Results The experimental results demonstrate that the proposed framework outperforms well compared to unimodal baselines, where CNN-based methods achieved 85% to 93% accuracy, while transformer-based pre-trained models reach 94.77%. In contrast, the proposed model achieves a peak accuracy of 96.87% accuracy, which highlights its effectiveness for skin disease classification. Discussion Unlike the traditional image-only system, the proposed model analyzed both visual features with their descriptive structured lesion descriptors that enhance interpretability and provide clinically meaningful diagnostic support.