Evaluation Artefacts in Deep Hate Speech Detection: A Reproduction Study and a Character-Aware Model Robust to Adversarial Obfuscation

Abstract

Automated hate speech detection is routinely reported at accuracies above 98%. We show that for two widely cited systems such figures are substantially artefacts of how their evaluation corpora were built, and we propose a model and protocol designed to avoid the same traps. First, we reproduce Ali et al. (2023) and Kibriya et al. (2024). Re-collecting the hashtag space of the former yields a corpus that is 91.3% duplicated; under the paper's stated protocol a lookup table that performs no learning scores 0.9804 accuracy, and deduplicating before splitting reduces the best model from 0.9915 to 0.5842. For the latter we rebuild the detection corpus from the paper's own recipe, matching its reported hate-class count exactly (22,862 tweets), and show that because the corpus merges two sources along the label axis, corpus identity nearly determines the label: the two sources are 98.8% separable from text alone, a provenance-only baseline reaches 0.8837 against a reported 0.933, and cross-corpus transfer falls below the target corpus's majority-class baseline in both directions. Second, we present a compact character-aware model a character CNN and corpus-trained WordPiece embedding feeding a BiLSTM with additive attention and a per-document graph convolutional branch evaluated over three seeds on a single human-annotated corpus with deduplication and an annotator-column guard. It attains 0.7354 macro-F1 (mean of three seeds) with 713k parameters, exceeding faithful re-implementations of both published architectures by 0.047 and 0.056 macro-F1 at a fraction of their size. Under five character-level evasion attacks it loses 0.252 macro-F1 against 0.359-0.379 for the neural baselines. We further report a quantitative evaluation of SHAP, LIME and attention against a randomdeletion control, finding all three faithful but attention clearly the weakest. We report two negative results: the graph branch's contribution lies inside the seed noise, and a TF-IDF linear baseline matches the proposed model on clean data and degrades comparably under attack. All code, splits and artefacts are released.