This study tests whether AI predictions can improve the statistical efficiency of social surveys when combined with census-scale auxiliary information and finds that supervised predictors yield effective-sample-size gains of about 7-12 percent when estimating the mean, while LLM gains are smaller but positive.
Abstract
LLMs can predict human survey responses with nontrivial accuracy. What this means for the future of social surveys remains hotly contested. Optimists anticipate a coming era of synthetic respondents; skeptics see a fragile and overhyped substitute. This article advocates a position: AI predictions cannot replace surveys, but they can augment them. Using prediction-powered inference (PPI), a statistical framework rooted in survey methodology and known in sociology as"mixed subjects design", the study tests whether AI predictions can improve the statistical efficiency of social surveys when combined with census-scale auxiliary information. Applying the framework to 83 social outcomes in the General Social Survey (GSS) and National Health Interview Survey (NHIS), with American Community Survey (ACS) being the auxiliary file, we find that supervised predictors yield effective-sample-size gains of about 7-12 percent when estimating the mean, while LLM gains are smaller but positive.
Large language models (LLMs) increasingly answer surveys on behalf of real panelists, and each panelist's recorded demographic attributes (e.g., age, education, ethnicity) are often placed in the prompt on the premise that richer profiles make more faithful synthetic respondents. While data minimization limits personal...
Machine learning (ML)-based nonresponse prediction in panel surveys enables selective interventions. However, the optimal use of ML predictions in Adaptive Survey Design (ASD) remains uncertain. We propose a method that integrates field experiment results on incentives, questionnaire length, and questionnaire content w...
John J. Collins, Saskia Bartholomäus, Tobias Gummer et al.· Sociological Methods & R...· 0 citations
Survey simulations require realistic combinations of answers, but predictive accuracy alone does not establish that a model can generate them. We evaluate the option probabilities returned by Jev, a TypeSafe decision model, using linked responses to 20 binary items from five archived survey waves. We compare population...
Conditioning language models on persona profiles is standard practice for generating synthetic survey responses, yet the relative contribution of different attributes to the fidelity of synthetic survey responses has not been systematically quantified. To test this, we conditioned four large language models (LLMs) on p...
Seoyoung Kim, Seonju Seo, Joseph Nak-Hyung Song et al.· 0 citations
When using a large language model (LLM) to simulate a user population, how should we combine individual behavioral histories to predict the fraction of users who take a particular action? We study this question in e-commerce, predicting observed product-level conversion rates from browsing sessions without task-specifi...
Comparing three model families and multiple math benchmarks, COMPASS outperforms the activation-steering baselines the authors compare against, improves GSM8K accuracy by 16 percentage points on average, and approaches CoT accuracy with 20-70\% fewer generated tokens.
Pratyay Dutta, Kowshik Thopalli, V. Narayanaswamy· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.