Sequence Labeling in Urdu Social Media Texts: Data Annotation and Transformer-Based Deep Learning Models
TL;DR
A transformer-based hybrid architecture is proposed, XLM-R–CNN–BiLSTM–CRF, which integrates XLM-R embeddings for contextualized representation, convolutional neural networks (CNNs) for local feature extraction, bidirectional long short-term memory (BiLSTM) networks for sequential modeling, and a CRF layer for optimal sequence prediction.
Abstract
Sequence labeling tasks such as part-of-speech (POS) tagging and named entity recognition (NER) have advanced significantly in high-resource languages and well-structured texts. However, low-resource languages, such as Urdu, face unique challenges due to limited annotated resources, complex morphology, and the informal, noisy nature of social media text. To address this, we introduce two relatively large-scale annotated datasets of Urdu tweets: a POS tagging dataset with 39 syntactic tags and an NER dataset covering three major entity types (person, location, and organization), capturing the linguistic diversity of user-generated content. We benchmark these datasets using a spectrum of approaches, from traditional conditional random fields (CRFs) to deep neural architectures with static embeddings and fine-tuned pre-trained language models (PLMs). Building on these findings, we propose a transformer-based hybrid architecture, XLM-R–CNN–BiLSTM–CRF, which integrates XLM-R embeddings for contextualized representation, convolutional neural networks (CNNs) for local feature extraction, bidirectional long short-term memory (BiLSTM) networks for sequential modeling, and a CRF layer for optimal sequence prediction. Our approach achieves state-of-the-art performance, with F1 scores of 95.39% for POS tagging and 91.91% for NER, significantly surpassing strong baselines. These resources and methods advance sequence labeling for Urdu while providing insights for other low-resource, noisy languages.