Piercing a Methodological Bubble: Large Language Models and the Social Sciences

Abstract

Large language models (LLMs) are increasingly used for text classification, survey simulation, and causal analysis in the social sciences. Yet many applications make a category error by treating systems trained to generate plausible language as if they directly measure social reality. This article argues that the problem is conditional rather than categorical. Autoregressive LLM outputs can have a definite relationship to social phenomena only when structured human involvement grounds, verifies, and anchors them to observable evidence. Without such grounding, they remain probabilistically plausible continuations shaped by linguistic patterns rather than by the political world itself. This distinction matters across use cases. Zero-shot generative coding and synthetic response generation are epistemically weakest; treating model outputs as observed variables creates similar problems. Supervised fine-tuned classification is sounder because human-labeled data provides an explicit connection, though the grounding comes from the labeling process rather than from the model alone. Recognizing this clarifies where LLMs are most useful: as tools for abductive reasoning and theory development, not autonomous instruments of measurement or inference in social science research.