Predict Me if You Can: Simulating User Responses with Synthetic Data: Evaluating the Fidelity of LLM-Based Synthetic Respondents
Abstract
Conditioning language models on persona profiles is standard practice for generating synthetic survey responses, yet the relative contribution of different attributes to the fidelity of synthetic survey responses has not been systematically quantified. To test this, we conditioned four large language models (LLMs) on profiles of 258 employed South Korean adults surveyed on AI-related job replacement risk. We grouped 11 attributes into three blocks (demographics, socioeconomic position & ideology, experience) and evaluated four cumulative levels of persona richness. A leave-one-out ablation identified the most impactful block. Fidelity generally rose with richer personas, and ablation showed that only experience-related attributes contributed meaningfully, while demographic attributes were net-neutral at best. For human-agent interaction (HAI) research with synthetic respondents, fidelity hinges on attribute relevance, not information quantity — neither the largest model nor all attributes maximize it.