This system for the Narrative Similarity task at SemEval-2026 (Task 4), where the goal is to determine which of two candidate stories is more similar to an anchor story directly or via vector representations, finds that chain-of-thought–style prompting with detailed reasoning outputs achieves comparable results to the scoring approach on difficult examples.
Abstract
We describe our system for the Narrative Similarity task at SemEval-2026 (Task 4), where the goal is to determine which of two candidate stories is more similar to an anchor story directly (Track A) or via vector representations (Track B). For Track A, our strategy leverages commercial, closed-source Large Language Models (LLMs) to generate multiple independent judgments per story triple. Simple majority voting provides strong performance in high-agreement cases, but it is unreliable when the judgments exhibit weak agreement. For difficult cases, we compare the stories along three narrative dimensions—theme, course of action, and outcome—prompting the LLMs to score similarity for each component on a scale of 1–4 and learning optimal combination weights on development data. We further find that chain-of-thought–style prompting with detailed reasoning outputs achieves comparable results to the scoring approach on difficult examples. We also conduct a data analysis revealing substantial annotation variability, which helps understand the difficulty of the task. Our system ranked 1 st in both tracks, achieving 0.78 test accuracy in Track A and 0.72 in Track B, where embedding only the course-of-action component yielded the best result.
Narrative similarity detection has broad applications in plagiarism detection, content recommendation, and comparative narrative analysis. We present a training-free, prompting-only framework for SemEval-2026 Task 4 (Track A), which requires identifying which of two candidate stories is narratively more similar to a given anchor story. Without any fine-tuning or additional annotations, we systematically evaluate three prompt templates across five structural prompting strategies, including zero-shot and few-shot inference, narrative summarization, keyword extraction, aspect splitting, and pair-wise comparison. Structured prompt templates and decomposed pairwise comparisons consistently outperform baseline configurations, achieving a peak accuracy of 72.50% on the test set and 67.75% on the final leaderboard (23th out of 44 teams).
Maximilian Schmerle, Nils Constantin Hellwig· SemEval@ACL· 1 citation
We present NarSiL 1 ( Nar rative Si milarity L earners), our system for SemEval-2026 Task 4 Track A on Narrative Story Similarity. NarSiL employs a two-stage architecture: a Mixture-of-Experts (MoE) initial classifier that also leverages supermajority voting across three large language models (Gemma-3-12B, GPT-3.5-turbo-instruct, and Gemini-2.5-Flash) over multiple runs, followed by a structured three-pathway fallback for ambiguous cases. The three pathways correspond directly to the task’s three core similarity components, abstract theme, narrative outcome, and course of action. Each path yields a similarity score corresponding to its respective component, and the scores are then combined through a weighted aggregation step. NarSiL achieves 64.25% accuracy on the official test set. An improved score of 70.25% is obtained by considering only the supermajority voting of GPT, followed by the previously described fallback.
B. Grecu, Costin-Gabriel Chiru, O. Cocarascu· SemEval@ACL· 1 citation
The Narrative Story Similarity and Narrative Representation Learning (NSNRL) task measures the narrative similarity between two stories based on three core aspects: the abstract theme, the course of action, and the outcomes. Our system leverages LLMs both for extracting high-level aspects and to encode them with state-of-the-art generalist embedding models. We then apply a series of embedding post-processing steps and learn to fit the embedding space with a Mahalanobis-like diagonal metric. We show that some of these techniques should not be applied universally, as they do not necessarily increase performance or overfit, depending on the base encoder. Our system outperforms the baseline only in Track B, ranking twelfth out of twenty-seven on the final leaderboard, while performing lower than the baseline accuracy in Track A.
Marius Marogel, Marius Popescu· SemEval@ACL· 1 citation
This paper presents Team HausaNLP’s sub-mission to SemEval-2026 Task 4 (Track A), which requires identifying the more narratively similar of two candidate stories relative to an anchor. Narrative similarity is defined along three dimensions: abstract theme, course of action, and story outcomes. We conduct a systematic ablation comparing five approaches: a lexical TF-IDF baseline, two bi-encoder SBERT variants ( all-MiniLM-L6-v2 and all-mpnet-base-v2 ), a paraphrase-focused embedding model, and a cross-encoder re-ranker. On the 200-instance development set, all-mpnet-base-v2 achieves the best performance (61.5% accuracy, 61.48 macro-F1), out-performing both TF-IDF (54.5%) and the official SBERT baseline (55.0%). Surprisingly, the cross-encoder re-ranker (55.5%) does not improve on the bi-encoders, which we attribute to the long-document nature of Wikipedia story summaries exceeding the model’s effective context window. On the official test set, our primary SBERT MiniLM submission achieved 61.50% accuracy (33rd of 44 teams). Our error analysis over 200 development instances identifies five systematic failure categories, distinct from the All Correct / Partial cases, including 23 Lexical Trap cases, 23 Hard Cases, and 24 Proposed-Recovery cases, thereby informing concrete directions for future work.
F. Adam, L. Aliyu, Sani Aji· SemEval@ACL· 1 citation
Narrative similarity assessment requires models to reason beyond surface-level lexical overlap and capture higher-level plot structures and thematic relationships. In this paper, we address SemEval-2026 Task 4 Track A: Narrative Story Similarity by reformulating it as an instruction-following generation problem. We employ parameter-efficient fine-tuning via LoRA to adapt pretrained large language models for triplet-based narrative comparison. To overcome the limitations imposed by the scarcity of human-annotated data, we further incorporate organizer-provided synthetic triplet samples generated by a large language model for data augmentation. Experimental results demonstrate that our fine-tuned Qwen2.5-7B model achieves slightly better performance than the zero-shot GPT-4o-mini base-line. These findings underscore the effectiveness of task-specific adaptation combined with synthetic data augmentation for narrative similarity modeling.
Haotan Guo, Hongbin Na, Zimu Wang et al.· SemEval@ACL· 1 citation
Here, we report our system for SemEval-2026 Task 5 (Gehring et al., 2026), which predicts graded plausibility scores for target word senses in narrative context. We explore embedding-based similarity, transformer fine-tuning, and a three-stage curriculum combining WiC pretraining, Wasserstein distribution learning, and KL-based calibration. Our best model, DeBERTa-xLarge with curriculum training, achieves 78% accuracy within one standard deviation and a Spearman Correlation of 0.70 , with an overall test score of 0.74. Results show that distribution modeling better aligns with human plausibility judgments than single-score prediction.
Fakeha Faisal, Rubab Shah, S. Zaidi et al.· SemEval@ACL· 1 citation· ⚡1