Toward Solving the Learning Rate Problem in Convolutional Neural Networks Using an Enhanced Cyclical Learning Rate Strategy
TL;DR
The Cyclical Local Adaptive Learning Rate Strategy (CLARA) is proposed, an enhanced CLR that introduces scheduler-level adaptive learning rate adjustment and dynamically adjusts the lower learning rate bound in response to training behaviour to improve convergence consistency during training.
Abstract
Convolutional neural networks (CNNs) are widely used for image segmentation and classification due to their strong ability to learn spatial representations. However, optimization stability and convergence efficiency remain challenging, particularly for tasks involving unclear or overlapping boundaries. Predefined learning rate schedules, such as cyclical learning rate (CLR), are commonly used in CNN training but offer limited adaptability to training dynamics. This study proposes the Cyclical Local Adaptive Learning Rate Strategy (CLARA), an enhanced CLR that introduces scheduler-level adaptive learning rate adjustment. CLARA dynamically adjusts the lower learning rate bound in response to training behaviour to improve convergence consistency during training. Integrated with U-Net, CLARA was evaluated on overlapping chromosome segmentation and achieved an Intersection over Union of 99.98% with 99.99% validation accuracy, while also producing more consistent segmentation outputs and improved separation in overlapping regions. Experiments on wafer defect classification using ResNet-50 achieved 99.39% validation accuracy, suggesting the applicability of the proposed scheduler across the evaluated segmentation and classification tasks.