Skip to content

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Sep 2026 · 0 citations · 26 references
Computer Science

TL;DR

A semantic correctness taxonomy is introduced that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content and CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI.

Abstract

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.

View source

Similar papers

Semantic Uncertainty Quantification Needs Factual Equivalence

A single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models.

Joseph Hoche, Quentin Guimard, Gianni Franchi · 0 citations
#artificial intelligence Preprint Oct 2026

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively target single answer retrieval. We introduce ListQA, a benchmark of 9,045 human-curated, cross-validated questions (audited error rate of 1.7%) that require LLMs to recall...

Mayank Kulkarni, Rakesh Vaideeswaran · 0 citations
Open access 2026

When Consistency Lies: Evaluating Prompt Paraphrasing for Black-Box Detection of Incorrect LLM Answers

Can a model’s answers to repeated factual questions help identify when it is wrong? A wrong answer may be repeated consistently, so this study compared two black-box signals: answers to identical prompts and answers to meaning-preserving paraphrases. A fixed sample of 150 questions not used in the feasibility pilot fro...

Jaineel Varshney, Bhavya Donda · 0 citations

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

Analysis reveals that scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning do not resolve these failure modes, demonstrating that they represent fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings rather than artifa...

He-Yuan Huang, Ji Dai, Alexandra DeLucia et al. · 0 citations
Book Open access Nov 2026

GQA-Q2Q: A Large-scale Benchmark for Entity Ambiguity Resolution in Visual Question Answering

Vision-language models achieve remarkable results on various visual question-answering benchmarks. However, they struggle with ambiguous questions where the target entity is unclear. To address this issue, it is essential to construct a benchmark dataset that pairs ambiguous questions with entity-grounded clarifying su...

Gyu-Min Park, Seong-Bae Park · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.