Evaluating Synthetic Survey Responses with Longitudinal Panel Data

Abstract

Interest in using large language models to generate synthetic survey responses has grown as researchers seek faster, scalable approaches to survey data collection. However, less is known about whether longitudinal probability-based panels improve synthetic-response predictions and whether improvements in predictive accuracy translate into more precise survey estimates. Using data from AmeriSpeak©, NORC’s probability-based panel, we conduct two evaluations spanning 47 questions, over 5 years and across 5,400 panelists. First, we compare configurations of retrieval-augmented generation (RAG) that supply models with respondents’ prior answers, responses to related questions, and relevant subgroup distributions. Second, we leverage the mixed-subjects design (MSD) to assess whether synthetic predictions can increase effective sample size when combined with observed human responses. We find that retrieving respondents’ own response histories increases prediction accuracy by 10 to 20 percentage points relative to a demographic-only baseline, substantially more than retrieving subgroup distributions alone. Prediction performance varies considerably across questions and is higher for items with fewer response options, lower entropy, greater response imbalance, and more stable response patterns. These predictive gains, however, rarely produce meaningful improvements in inferential precision. For example, supplementing a human sample with a synthetic sample twice its size yields little or no gain in effective sample size for most questions. Benefits are concentrated among a small number of stable, highly predictable outcomes and scenarios with small human samples. The findings demonstrate the value of longitudinal human data for improving synthetic response modeling while underscoring the need to evaluate predictive and inferential performance separately. Synthetic responses may supplement human data in selected applications, but the results do not support their general use as substitutes for probability-based survey data.