Context-Enhanced Multimodal Learning Framework for Effective Hateful Meme Detection

Abstract

Hateful memes carry harm through the complex combination of image and text, requiring a detector to analyze both modalities jointly. This study investigates whether a short, text-aware semantic description of each meme, generated once by a small vision-language model, adds complementary information beyond direct image-text cross-attention. Qwen2-VL-2B generates a two-to-three sentence description for every meme based on its visual content and overlaid text. Frozen CLIP and RoBERTa encode the visual, textual, and semantic description streams, while a trainable fusion head integrates them via bidirectional cross-attention with a learned gating mechanism. We evaluate our framework on the Facebook Hateful Memes dataset (9,000 labeled memes) using group-aware five-fold cross-validation across 5 random seeds. Across the Facebook Hateful Memes benchmark, the Qwen2-VL-augmented architecture yields an average AUROC reaching 0.8180 alongside a 0.6575 F1-score, noticeably surpassing both uncaptioned baselines (0.7230 AUROC) and control setups using permuted descriptions (0.7995 AUROC) Furthermore, evaluation across additional datasets—MAMI (Misogyny) and HarMeme (Social/COVID Harm)—demonstrates strong out-of-domain generalization (0.7680 and 0.7980 AUROC, respectively). Description-removal ablations confirm that intermediate semantic interpretations bridge subtle visual-textual incongruities, while cross-attention gating selectively filters description noise.