How Reliable Is LLM Confidence in Hindi and Marathi? A Study of Calibration and Answer Stability

Abstract

Language models are increasingly used to answer exam-style questions in Indian languages, and their confidence scores are often treated as a signal for when an answer can be trusted. We checked how far that trust is justified for two small open-weight models, Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct, using 300 native-language multiple-choice questions each in Hindi and Marathi from the MILU benchmark. Confidence was read directly from the probability the model assigns to each answer letter. The 3B model scored 45.0% in Hindi and 32.7% in Marathi while reporting an average confidence above 91%, which gives expected calibration errors (ECE) of 0.483 and 0.592. The Marathi–Hindi difference in ECE had a 95% bootstrap interval of 0.034 to 0.185. Reordering the answer options, a change that leaves the question unchanged, altered the chosen answer for about 38–40% of questions with the 3B model and 52–57% with the 1.5B model, even though overall accuracy and calibration hardly moved. A single temperature parameter fitted on half of the questions brought held-out ECE of the 3B model below 0.1 in both languages, including on reordered options, but it did not improve the model’s ability to separate right answers from wrong ones (AUROC about 0.6). This is a small-scale study of one model family and one prompt format, and we state its limits openly. The main message is practical: an overall accuracy number can look steady while individual answers and confidence scores are not. Keywords: calibration; large language models; Hindi; Marathi; prompt sensitivity; selective prediction; temperature scaling