Skip to content
Open access

Human vs LLM in Conversational Repair Annotation: A New Resource and Comparative Study

2026 · International Conference on Language Resources and Evaluation · pp. 6880-6892 · 1 citation · 28 references
Computer Science

TL;DR

This work introduces OIR annotations for the NOXI corpus, achieving considerable reliability and evaluating whether LLMs can reliably annotate OIR sequences using structured Chain-of-Thought prompting and comparative analysis across two corpora, finding weak alignment between LLMs and human annotations, particularly in recognizing trouble signaling.

Abstract

Addressing the scarcity of annotated data for Other-Initiated Repair (OIR), when recipients interrupt conversation progressivity to signal trouble, prompting speakers to provide repair, this work introduces OIR annotations for the NOXI corpus, achieving considerable reliability. We evaluate whether LLMs can reliably annotate OIR sequences using structured Chain-of-Thought prompting and conduct comparative analysis across two corpora: NOXI (natural dialogue) and CABB-S (Dutch, task-oriented), finding weak alignment between LLMs and human annotations, particularly in recognizing trouble signaling. Analyzing human-LLM disagreement using the LLM-generated explanations revealed limitations: models rely on lexical patterns rather than conversational context, construct reasonable-sounding but misleading narratives, highlighting crucial limitations for both automated annotation of complex interactional phenomena.

Read PDF

Similar papers

Preprint Oct 2026

Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs

We propose a methodology for investigating the differences between natural and synthetic language in a challenging setting: humorous improvisational dialogue. Our experimental setting compares human-produced comedy improvisations with the dialogues generated by LLMs in a multi-agent environment that we designed. The an...

Lizeth Palacios Patiño, Ximena Gutierrez-Vasques, Daniel Rodriguez · 0 citations
Open access Sep 2026

Context Projector: Complementary Keyword and Dialogue Context Embeddings for LLM-based ASR

Contact-center dialogue systems require both accurate transcripts and reliable recognition of business-critical entities. We introduce a hybrid keyword and context projector (CP) approach for LLM-based ASR. Rather than appending raw dialogue history, CP encodes previous turns into compact contextual tokens, complemente...

Esaú Villatoro-Tello, Sergio Gastón Burdisso, Shashi Kumar et al. · 5 citations

Clarification Is Not Correction: LLMs Fail to Let Go

Research for interactive LLMs from retaining more context toward preserving uncertainty is redirected from retaining more context toward preserving uncertainty, to motivate uncertainty-preserving state management.

Jian-Zhe Lin, Xiao-Lin Li, Fei Wang et al. · 0 citations
#artificial intelligence Open access Oct 2026

SimXAI: A Simple Drop-in Layer Increases Interaction Quality and Understanding in Conversational XAI

Conversational explainable AI (ConvXAI) agents often use GUS-style architectures with separate modules for intent recognition, dialogue management, and template-based response generation. While faithful to the underlying XAI methods, these pipelines are hard to adapt and often struggle with context-dependent follow-up...

Van Bach Nguyen, Qianli Wang, Nils Feldhus et al. · 0 citations
#artificial intelligence Preprint Nov 2026

Where Did the Evidence Go? Diagnosing Proof-Set Loss in Long-Term Conversational Memory

Long-term conversational question answering can require several separated dialogue turns that support an answer only together. We introduce EviCover, a retrieval approach designed to retain this complementary evidence. It converts questions into observable evidence requirements, checks requirement coverage using retrie...

Bogdan Roshchupkin, M. Khodorchenko · 0 citations

Controlled Attribute-Specific Summarization of Interrogative Dialogues

Effective summarization of interrogative dialogues is a critical task in forensic and investigative settings, requiring high factual accuracy, coherence, and attribute-specific relevance. In this work, we introduce CASPER, a novel Chain-of-Thought Attribute-Specific Prompting for Evaluative Summarization framework that...

A. Bhardwaj, Arjit Singh Arora, Md. Shad Akhtar · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.