Confidence and Correctness Across Synthetic Persona Prompt Conditions in a Large Language Model

Abstract

This study investigates whether assigned synthetic persona prompt labels are associated with correctness and expressed certainty in a large language model. A 4 × 2 × 3 × 3 factorial design generated 72 synthetic persona conditions for Gemini 1.5 Flash. All 16 archived question-response CSV files were audited; seven distinct tasks with assigned answer keys were included in the primary accuracy endpoint, and nine were excluded because they were subjective, ambiguous, or duplicated an archived task. Recalculation from 504 responses found 459 correct (91.07%). Mean reported certainty on the seven scored tasks was 97.0%. At the persona level, categorical-factor ordinary least squares models with HC3 heteroskedasticity-robust standard errors were refit to the new seven-item accuracy and c ertainty o utcomes (accuracy R² = 0 .585; c ertainty R² = 0 .246). C oefficients a re e xploratory, n ot causal, and question-level accuracy is presented descriptively where all-correct or sparse cells preclude stable logistic inference. The study demonstrates the importance of explicit scoring, categorical coding, and separating observed model confidence from validated probability calibration. Sensitivity analysis excluding two base-rate items produced 347/360 (96.39%) and did not retain the primary political-view association; therefore, apparent persona effects should be treated cautiously.