M-NLE: Knowledge-Augmented Multitask Learning for Offensive Meme Detection and Explanation Generation

Abstract

Memes combine visual content and short textual cues to convey humor, sarcasm, and social commentary, but they can also encode offensive or harmful messages targeting individuals and social groups. While recent multimodal systems have improved offensive meme detection, many of them operate as opaque classifiers and do not explain why a meme is considered offensive. This limits their usefulness in moderation settings, where human reviewers and policy stakeholders often require an interpretable account of the harmful implication. In this paper, we present M-NLE, a compact knowledge-augmented multitask model for jointly detecting offensive memes and generating natural language explanations. M-NLE retrieves external contextual knowledge in the form of visual entities and their descriptions, incorporates this knowledge through a dedicated knowledge encoding module, and uses the resulting multimodal representation for both offensiveness classification and explanation generation. Experiments on the extended Facebook Hateful Memes and MultiOFF datasets show that M-NLE achieves competitive classification and reference-based generation performance across multiple data partitions while remaining substantially smaller than general-purpose LLM/VLM systems. Human evaluation shows that M-NLE improves explanation adequacy and fluency over the evaluated baselines while also highlighting areas for further improvement. Overall, 38% of the generated explanations were rated as not related to the input, whereas 15% were judged to fully justify the reference interpretation. These findings suggest that M-NLE produces more useful and fluent explanations than existing baselines, while leaving room to further improve explanation relevance and justification. Ablation studies further show the contribution of knowledge encoding and multitask learning, while detailed qualitative analyses reveal both the benefits and limitations of retrieved knowledge, including cases where it introduces noise or bias. Overall, our results suggest that knowledge-augmented explanation generation is a practical direction for more interpretable offensive meme detection. To facilitate reproducibility, we publicly release the complete implementation, experimental scripts, and detailed documentation at https://github.com/newcodevelop/M-NLE