A systematic ablation study of three multilingual transformer architectures evaluated on HeteroSarc-47K, a curated multi-domain dataset containing 47,694 instances across six heterogeneous domains, highlighting that class imbalance remains a critical challenge independent of architecture.
Abstract
Sarcasm detection remains a challenging task in natural language processing due to the complex interaction between linguistic and pragmatic cues. While transformer-based models have shown encouraging performance, it is not yet clear whether improvements stem primarily from architectural design or explicit feature engineering. This work introduces a systematic ablation study of three multilingual transformer architectures (mBERT, mDeBERTa-v3, and XLM-RoBERTa) evaluated on HeteroSarc-47K, a curated multi-domain dataset containing 47,694 instances across six heterogeneous domains, including social media and dialectal Arabic text. We isolate the empirical impact of two feature augmentation techniques: emoji integration and contrastive pre-training. Our experiments reveal that while contrastive pre-training increases sarcasm recall for mBERT, it induces a significant drop in its overall accuracy, demonstrating architecture-specific sensitivities. In contrast, architectural upgrades alone consistently yield robust improvements; mDeBERTa-v3 achieves the highest performance (72.79% sarcasm recall and 87.20% accuracy) without requiring any feature augmentations, outperforming the baseline mBERT. Furthermore, per-class analysis reveals a persistent performance gap between non-sarcastic and sarcastic classes across all domains, highlighting that class imbalance remains a critical challenge independent of architecture. Ultimately, these findings provide a vital practical takeaway for practitioners: upgrading to inherently stronger multilingual architectures yields significantly higher and more robust performance returns compared to investing in extensive feature-engineering pipelines.
Sarcasm detection is a complex task in natural language processing because it depends on implicit mood variations, contextual comparison, and the congruence of polarity between the literal form of expression and its intended meaning. Although the transformer-based models, including BERT and DeBERTa, have been taking a great leap in performance by introducing contextual self-attention, they mainly learn sarcasm patterns with an implicit hypothesis of sentiment polarity contradictions, which define the discourse of sarcasm. In this paper, a polarity-sensitive transformer model that explicitly incorporates sentiment data in representation learning is presented to detect sarcasm. In contrast to traditional fine-tuning methods, which treat sarcasm as a generic classification problem, the methodology adds sentiment-polarity cues to the embedding space, enabling the model to fine-tune contextual representations in a polarity-sensitive way. The proposed method will improve the ability of semantic representations of a context to reflect incongruity patterns in contextual segments. The positive results of experiments on benchmark sarcasm datasets indicate that explicit polarity integration is more robust and generalizes better than traditional transformer baselines, particularly in context-specific situations. The findings indicate that embedding-level sentiment improvement offers a sound theoretical and practical guideline in the process of expanding sarcasm detection beyond implicit contextual modeling.
S. Nagini, K. Akhil, Harshitha Upadhyayula et al.· International journal of com...· 0 citations
Test set results show that Decoding-Enhanced Bert with Disentangled Attention (DeBERTa) achieves the highest macro F1 − Score of 85.48%, surpassing the previously top-ranked Multi-Task Learning (MTL) system, which attains a macro F1 of 83.07%.
Batyr Sharimbayev, S. Kadyrov· Journal of Advances in Infor...· 0 citations
This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-English code-switched discourse. It evaluates MARBERT and XLM-R(oBERTa), with BERT serving as a general-purpose baseline. The models were evaluated on their ability to classify context-sensitive pragmatic functions in code-switched social-media discourse. 11695 unique X posts were collected via Python and utilized for the study. The study employs a quantitative and qualitative NLP approach, following a supervised pipeline. Findings unveil that MARBERT consistently surpasses XLM-R with validation Macro F1 increasing from 0.39 to 0.84 and validation loss decreasing from 0.55 to 0.19. On an independent test set, it achieved 0.96 accuracy, 0.83 macro precision, 0.87 macro recall, and 0.85 Macro F1, while XLM-R achieved 0.92 test accuracy but a substantially lower Macro F1 of 0.52. This was also supported by class-level performance where MARBERT outperforms XLM-R considerably with F1 improvements ranging from +0.33 to +0.60, demonstrating a clear advantage in modeling Arabic digital pragmatics. The study concludes that Transformer performance depends more on DSPP than multilingual coverage alone, as the latter does not guarantee optimal performance on a highly specialized pragmatic classification task.
Fahad Saud Al Hussen, King Saud University, Riyadh et al.· 0 citations
A multilingual medical VQA benchmark over eight languages is constructed, organized into four representative scenarios that isolate the core capabilities medical VQA requires, and a training-free scenario-aware representation engineering method is proposed, leveraging LVLMs's superior English medical VQA capability to steer non-English representations toward their English counterparts at inference time.
Jingbo Wang, Sendong Zhao, Haochun Wang et al.· 0 citations
The results show that hybrid ensemble methods can improve token-level accuracy in low-resource POS tagging, while also revealing a trade-off between frequent-tag accuracy and rare-tag robustness.
Abdelouahed Moussaoui, Nor-Eddine Azalmad, Said Bahassine et al.· Information· 0 citations
Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored, particularly for relatively free word-order languages. We investigate the structural sensitivity of multilingual LLMs using two linguistically grounded perturbation settings in Hindi and Malayalam: constrained constituent reordering and active-passive voice transformation. We introduce a benchmark dataset IndicReStruct, with two variants, GSM8K-Reordered and GSM8K-Voice, constructed from GSM8K while preserving semantic meaning. Across six state-of-the-art LLMs and multiple prompting strategies, we observe consistent and significant degradation in mathematical reasoning performance under structurally perturbed inputs. To further understand these failures, we perform qualitative error analysis and mechanistic interpretability experiments using residual-stream activation patching. Our analyses show that reasoning failures frequently arise from disruptions in entity-quantity alignment and that intermediate transformer layers contribute most strongly toward reasoning restoration. Overall, our findings suggest that current multilingual LLMs remain highly sensitive to surface syntactic realization and lack robust compositional invariance under structurally different but semantically equivalent inputs.
Karthika Nhayakkat, Rajat Verma, Maharaj Brahma et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
