Real-time hallucination detection and intervention in medical LLMs via calibrated hidden-state probes
TL;DR
A calibrated hidden-state probing pipeline that enables token-time hallucination detection under explicit false-positive-rate (FPR) constraints, making it suitable for streaming clinical deployment.
Abstract
Objective
Most hallucination mitigation for large language models (LLMs) operates post-hoc, leaving safety-critical clinical deployment without real-time warning capability. We present a calibrated hidden-state probing pipeline that enables token-time hallucination detection under explicit false-positive-rate (FPR) constraints, making it suitable for streaming clinical deployment.
Methods
We formulate token-time hallucination detection as a constrained sequential decision problem separating non-circular supervision construction (using ROUGE-L, exact-choice, and exploratory NLI verifiers), FPR-constrained operating-point selection, and downstream intervention evaluation into a modular, reusable protocol. A lightweight two-layer hidden-state probe classifies each generated token into one of three risk states (Safe, AtRisk, or Hallucinating), where the intermediate AtRisk state captures pre-error instability. We validate across four medical QA benchmarks (Endoscopy, PubMedQA, MedHallu, MedQA-USMLE) spanning approximately 44,500 generation trajectories and three backbone LLMs (Qwen3-8B, Llama-3.1-8B-Instruct, BioMistral-7B).
Results
On the public biomedical QA benchmarks the probe yields viable detection across all backbone-dataset combinations: EDR@5 up to 0.390 (PubMedQA) and 0.352 (MedHallu) at the strict cap γ≤0.10, rising to 0.587 with 29%-59% hallucination reduction at the monitoring cap γ≤0.20. The trigger fires roughly 10-35 tokens before error onset, whereas a post-hoc check has zero lead time by construction. A single-institution Endoscopy corpus serves as a case study (EDR@5 0.613, Llama-3.1), treated as illustrative because its small confirmed-correct denominator (26-52 per backbone) yields wide confidence intervals. Probe latency is ∼10ms per answer versus ∼10s for multi-sample baselines.
Conclusion
Hidden-state probing with FPR-constrained calibration provides a practical, low-latency solution for real-time hallucination monitoring in clinical LLM deployments. The modular pipeline separating supervision construction, probe training, and trigger calibration is directly reusable with alternative detectors or verifiers.