Comparative evaluation of CNN, transformer, and CLIP models for multiclass fruit classification in robotic sorting systems

Abstract

Fruit classification and automated sorting play a critical role in modern agricultural logistics and food processing, yet remain challenging due to complex visual variations in real-world scenarios. This paper uses a fruit classification system based on deep learning techniques and automated sorting system, leveraging three representative model architectures— Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Contrastive Language-Image Pre-training (CLIP)—to address multi-class fruit recognition. To ensure a fair and comprehensive evaluation, we use a multi-class fruit dataset comprising 3,326 images with diverse real-world variations, including varying lighting conditions, viewing angles, and background clutter, and train all models under a unified experimental setting. Extensive experiments reveal that CNN-based models achieve relatively lower classification performance, as they are limited by their focus on local feature extraction. In contrast, Transformer-based models improve accuracy by capturing long-range contextual dependencies through global feature modeling. Notably, the CLIP model with a ViT-B/16 backbone achieves the best overall performance, demonstrating exceptional generalization ability in complex visual scenarios. The experimental results confirm the effectiveness of pre-trained vision-language models for practical fruit classification tasks, and the proposed system is further validated through robotic arm sorting simulations, confirming its potential for real-world deployment in automated agricultural sorting pipelines.