Diffusion-Enhanced NAT–BART Vision Language Transformer for Unified Urdu Word Recognition

Abstract

Handwritten Urdu text recognition remains a very challenging problem due to the cursive and complex nature of the Nastaliq script. Each writer has a unique style, and there is a severe lack of large, well-labeled datasets. These challenges make Urdu text recognition much harder than for many other languages. Here we propose a novel and effective approach for handwritten Urdu word recognition that addresses both data scarcity and feature complexity. We utilized conditional diffusion models to generate synthetic handwritten Urdu word images. These models produce realistic word samples covering a wide range of variations, such as stroke thickness, curves, and writing styles. The synthetic images augment the training data, helping the recognition model learn better in low-resource settings. For recognition, we employ a Neighborhood Attention Transformer (NAT) to extract robust visual features from both real and generated images. The visual embeddings are then fed into a pretrained BART language model configured as an encoder–decoder, which predicts the correct Urdu word. In this work, we first construct a baseline handwritten Urdu word dataset, addressing the lack of publicly available dataset for the text recognition. Building upon this foundational dataset, we leverage diffusion based generative approach to generate diverse and realistic synthetic handwritten samples, significantly alleviating the challenges of manual data collection. To the best of our knowledge, this is the first study to introduce both a real handwritten Urdu word dataset and a diffusion-generated synthetic dataset. In addition, we develop a unified word recognition model trained jointly on handwritten and printed Urdu word data, leading to improved recognition robustness and performance.