Skip to content
Open access

When Consistency Lies: Evaluating Prompt Paraphrasing for Black-Box Detection of Incorrect LLM Answers

2026 · American Journal of Student Research · 0 citations

Abstract

Can a model’s answers to repeated factual questions help identify when it is wrong? A wrong answer may be repeated consistently, so this study compared two black-box signals: answers to identical prompts and answers to meaning-preserving paraphrases. A fixed sample of 150 questions not used in the feasibility pilot from SimpleQA Verified was given to GPT-5 Nano and Claude Haiku 4.5. Each model answered each question in five independent API calls: three identical original prompts and two paraphrases (1,500 responses total). Original Run 1 (O1) was the target answer. Repeat consistency was the mean pairwise semantic agreement among O1, O2, and O3; paraphrase consistency used O1, P1, and P2. Error scores were defined as 1 minus consistency. Discrimination was measured using receiver operating characteristic area under the curve (ROC-AUC), with 95% confidence intervals from 10,000 question-level bootstrap resamples. After excluding 31 O1 nonanswers, 269 cases remained (37 correct and 232 incorrect). Repeat-consistency AUC was 0.751 (95% CI, 0.671-0.824), compared with 0.714 (0.618-0.801) for paraphrase consistency; the difference was -0.037 (95% CI, -0.117 to 0.043). Paraphrasing was not shown to improve overall discrimination. Of 232 incorrect answers, 52 (22.4%) remained perfectly consistent across identical repeats, and paraphrasing exposed disagreement in 14 of these 52 (26.9%). Paraphrases therefore revealed some errors that identical repeats missed, although superiority as an overall screening measure was not established.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.