CNN-Transformer Based Deep Learning Framework for Hindi Handwritten Text Recognition

Abstract

There exist significant difficulties in recognizing lowresourced handwritten Indic language text due to the large variations in handwriting, complex structure, and insufficient annotated data. In this paper, we consider a Hybrid CNN-Transformer architecture for handwritten word recognition tasks using the IIIT-INDIC-HW-WORDS dataset for Hindi. ResNet50 encoder and Transformer decoder models have been employed for feature extraction and sequence generation, respectively. To promote faster convergence and enhance the generalization ability of the trained model, three different training phases involving gradual freezing and unfreezing layers were adopted. Additionally, CLAHE and elastic distortion have been proposed as pre-processing techniques for dealing with noise and variation in the Indic script. Lexicon postprocessing, which involves word correction using the edit distance search algorithm of a specific domain lexicon (11,031 words), has also been considered. The experimental results revealed that the character error rate was 3.07% using greedy decoding and 2.26% after lexicon correction giving 93% exact match accuracy. This represents a substantial improvement compared to the previous approach using the Hybrid CNN-Transformer based on ResNet18 with CER of 7.58% for Hindi text recognition tasks.