HaluCheck, a large-scale, bilingual benchmark comprising over 42k instances spanning both factual hallucinations in text and faithfulness hallucinations in agent trajectories, provides a solid basis for benchmarking and diagnosing LLM reliability across text and agent scenarios.
Abstract
Large language models (LLMs) are widely used for information access, content production, and agent-based task execution, but their outputs remain vulnerable to subtle, high-risk hallucinations. To systematically evaluate and advance post-generation hallucination detection, we introduce HaluCheck, a large-scale, bilingual benchmark comprising over 42k instances spanning both factual hallucinations in text and faithfulness hallucinations in agent trajectories. HaluCheck features fine-grained, multi-level annotations for detection, localization, and correction, covering different tasks and perturbation degrees. Experiments on fourteen LLMs reveal that, while models show basic sample-level detection ability, they struggle with fine-grained localization, correction, and agent hallucination detection. Further analysis demonstrates that thinking, web search, and supervised fine-tuning each offer partial improvements, but none fully solve the problem. HaluCheck thus provides a solid basis for benchmarking and diagnosing LLM reliability across text and agent scenarios.
Hallucination in large language models, the generation of fluent but factually unsupported text, remains a barrier to their safe deployment. Existing zero-resource detectors such as SelfCheckGPT measure consistency across stochastic samples but treat all claims uniformly and draw a fixed, often excessive, number of gen...
Tetala Rukmender Reddy, Vedant Mathur, Tanay Aggarwal et al.· 2026 6th International Confe...· 0 citations
This work proposes a five-step diagnostic protocol covering feature correlation, bootstrap stability, sparse versus dense ranking disagreement, intervention baselines, and cross-dataset evaluation as a minimum standard for sparse-neuron localization claims, and investigates prior work using the proposed approach.
Huseyin Cavus, Sebin Sabu, J. Spear et al.· arXiv.org· 0 citations
This research proposes a multi-signal framework for hallucination detection and mitigation that corrects up to 78.4% of the detected hallucinations and highlights that targeted, type-aware hallucination mitigation can significantly improve answer reliability while remaining computationally effective.
In this paper, we present an empirical analysis of hallucination behavior in small language models (1.5B–7B parameters), addressing the question of whether such failures are stochastic or structured. Our analysis of 4,500 responses from three instruction-tuned models, evaluated on TruthfulQA and HaluEval, reveals that...
Kushagra Agrawal, Yu-Ming Feng, Hang-Jun Che et al.· International Conference on...· 0 citations
The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth, enabling the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity.
Jia-Wei Dong, Zi-Long Bai, Peng-Tian Zhu et al.· 0 citations
Hallucinations are false or unverifiable outputs generated by Large Language Models (LLMs) that reduce their trustworthiness in critical applications such as healthcare, education, and research. This paper proposes a hybrid framework that unifies three complementary verification signals for the detection and reduction...
Radhika K. R., Sumit Gupta, Nagesh H. Shenoy et al.· 2026 Forth International Con...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.