Open access

Team HausaNLP at SemEval-2026 Task 4: Narratives via Semantic Embeddings

F. Adam L. Aliyu Sani Aji

2026 · SemEval@ACL · pp. 48-53 · 1 citation · 9 references

Computer Science

TL;DR

A systematic ablation comparing five approaches to identify the more narratively similar of two candidate stories relative to an anchor identifies five systematic failure categories, distinct from the All Correct / Partial cases, including 23 Lexical Trap cases, 23 Hard Cases, and 24 Proposed-Recovery cases.

Abstract

This paper presents Team HausaNLP’s sub-mission to SemEval-2026 Task 4 (Track A), which requires identifying the more narratively similar of two candidate stories relative to an anchor. Narrative similarity is defined along three dimensions: abstract theme, course of action, and story outcomes. We conduct a systematic ablation comparing five approaches: a lexical TF-IDF baseline, two bi-encoder SBERT variants ( all-MiniLM-L6-v2 and all-mpnet-base-v2 ), a paraphrase-focused embedding model, and a cross-encoder re-ranker. On the 200-instance development set, all-mpnet-base-v2 achieves the best performance (61.5% accuracy, 61.48 macro-F1), out-performing both TF-IDF (54.5%) and the official SBERT baseline (55.0%). Surprisingly, the cross-encoder re-ranker (55.5%) does not improve on the bi-encoders, which we attribute to the long-document nature of Wikipedia story summaries exceeding the model’s effective context window. On the official test set, our primary SBERT MiniLM submission achieved 61.50% accuracy (33rd of 44 teams). Our error analysis over 200 development instances identifies five systematic failure categories, distinct from the All Correct / Partial cases, including 23 Lexical Trap cases, 23 Hard Cases, and 24 Proposed-Recovery cases, thereby informing concrete directions for future work.

DOI Semantic Scholar

Read PDF

Similar papers

HausaNLP at SemEval-2026 Task 7: A Locale-Conditional, Multi-Model Prompting Pipeline for Hausa Cultural Question Answering

Error analysis shows that a non-trivial fraction of failures are placeholder strings caused by API errors rather than incorrect generations, and that surface-level mismatches (verbosity, ortho-graphic variation) account for many of the remaining errors.

F. Adam, Sani Aji, L. Aliyu et al. · 0 citations

Jul 2026

Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM shows that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.

S. Subramanian, M. Gungor, Vikram Elango · arXiv.org · 1 citation

Review Jul 2026

Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering

DCASE~2026 Task~5 introduces Audio-Dependent Question Answering (ADQA), which tests whether large audio-language models answer from the audio rather than from textual priors. An Audio-Dependency Filtering (ADF) pipeline combines silent-audio probing, per-option perplexity, a large language model (LLM) commonsense check...

Haolin He, Renhe Sun, Zheqi Dai et al. · 1 citation

#artificial intelligence Preprint Aug 2026

SemPOI-RL: Aligning LLM Semantic Reasoning for Interpretable Out-of-Town POI Sequential Generation

SemPOI-RL is proposed, a framework that aligns LLM semantic reasoning with structured sequence generation for interpretable OOT recommendation and consistently outperforms both traditional recommenders and direct LLM baselines, while providing interpretable style attribution across different phases of a trip.

Yunqi Liu, Yang Zhang, Ruixing Zhang et al. · 0 citations

Team hugang11 at SemEval-2026 Task 1: A CoT-SFT, Teacher-Constructed DPO, and Deterministic Post-processing Pipeline for Chinese Humor Generation

The hugang11 system addresses a practical trade-off in creative text generation: models that produce sharper and more stylized jokes often become less stable in output format, and builds a three-stage pipeline that combines chain-of-thought-augmented supervised fine-tuning (CoT-SFT), teacher-constructed direct preferen...

Gang Hu, Liu Yang, Jing Li · 1 citation

System Report for CCL25-Eval Task 4: Factivity Inference Based on Dynamic Few-Shot Learning

This paper presents the implementation approach used in the First Chinese Factivity Inference Evaluation 2025 (FIE2025), which aims to enhance LLM’s discriminative capability by adequately integrating the task-specific information via prompts, as well as constructing dynamic few-shot datasets for fine-tuning.

Sunyan Gu, T. Lu, Siqi Liu et al. · 1 citation