From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models

Abstract

Probe-based uncertainty estimation (UE) has emerged as a prominent approach to detect hallucinations in Large Language Models (LLMs) by learning uncertainty from internal model signals. Yet, recent methods vary simultaneously across feature design, training data construction, and evaluation setting, obscuring what actually drives performance. To address this issue, we propose a factorised study of probe-based UE under matched conditions. Our results show that raw hidden states and attention features are difficult to outperform in-domain. However, under distribution shift, structured and compressed features are more robust, suggesting that in-domain performance alone is insufficient to measure progress. Furthermore, prompting and label construction significantly affect probe behaviour. Building on these empirical design principles, we construct benchmark-pretrained probes that retain useful discriminative signal on open-ended factual generation. Their competitive performance relative to limited-label target setting probes and sampling-based estimators suggests that they can provide a practical starting point for subsequent target-domain adaptation. More broadly, our work encourages deployment-oriented evaluation of probe-based uncertainty estimators.