Parameter-Efficient Adaptation of Visual Foundation Model for Wafer Map Defect Classification

Abstract

As semiconductor manufacturing becomes increasingly complex, the fast and accurate classification of spatial failure patterns in wafer maps has become a key enabler for economic efficiency, supporting timely corrective actions in process control and equipment management. While Deep Convolutional Neural Networks (CNNs) establish the benchmark for wafer map defect classification, their dependence on large, manually annotated datasets severely constrains operational agility. The requirement of upwards of 15,000 labeled samples per deployment (as seen in the WM-811K benchmark) makes it highly impractical for fabrication environments to quickly adapt to new defect signatures. To address this, we propose a three-tier deployment framework built on visual foundation models (specifically DINOv2) that allows practitioners to flexibly trade annotation cost for accuracy. In Tier 1, we tackle the zero-backbone-update scenario. By introducing Patch+[CLS], a spatial aggregation strategy that concatenates the global classification token ([CLS]) with mean-pooled patch tokens, a frozen DINOv2 base model achieves 91.87% accuracy and 88.78% F1-Macro on WM-811K. When only a small support set is available within the existing defect taxonomy, Tier 2 uses a lightweight metric-learning projection head to reach 86.8% accuracy using only 10 support examples per class for prototype construction, enabling rapid deployment against already-known defect types. We further validate adaptation to genuinely novel defects under a strict class-incremental setting, in which a defect class is withheld entirely from training and later recognized from as few as one labeled example while the frozen backbone and projection head remain unmodified. For peak performance, Tier 3 pairs our Patch+[CLS] aggregation with parameter-efficient fine-tuning via low-rank adaptation (LoRA). By updating only 0.67% of the backbone parameters (0.59 M of 87.2 M), this approach achieves 95.32% accuracy and 93.47% F1-Macro. This strategy effectively surpasses a fully supervised ResNet-18 baseline without the need for custom, domain-specific architectures. Finally, we present a systematic ablation of self-supervised learning paradigms. Our results show that self-distillation objectives substantially outperform masked-reconstruction (MAE) and contrastive image-text (SigLIP) methods on industrial wafer map data. We also validate the robustness of the framework through multi-label detection on MixedWM38 (mAP = 0.996) and prototype-based open-set anomaly rejection (AUROC = 0.876).