Team HausaNLP at SemEval-2026 Task 4: Narratives via Semantic Embeddings

Abstract

This paper presents Team HausaNLP’s sub-mission to SemEval-2026 Task 4 (Track A), which requires identifying the more narratively similar of two candidate stories relative to an anchor. Narrative similarity is defined along three dimensions: abstract theme, course of action, and story outcomes. We conduct a systematic ablation comparing five approaches: a lexical TF-IDF baseline, two bi-encoder SBERT variants ( all-MiniLM-L6-v2 and all-mpnet-base-v2 ), a paraphrase-focused embedding model, and a cross-encoder re-ranker. On the 200-instance development set, all-mpnet-base-v2 achieves the best performance (61.5% accuracy, 61.48 macro-F1), out-performing both TF-IDF (54.5%) and the official SBERT baseline (55.0%). Surprisingly, the cross-encoder re-ranker (55.5%) does not improve on the bi-encoders, which we attribute to the long-document nature of Wikipedia story summaries exceeding the model’s effective context window. On the official test set, our primary SBERT MiniLM submission achieved 61.50% accuracy (33rd of 44 teams). Our error analysis over 200 development instances identifies five systematic failure categories, distinct from the All Correct / Partial cases, including 23 Lexical Trap cases, 23 Hard Cases, and 24 Proposed-Recovery cases, thereby informing concrete directions for future work.