Vibe Coding for Statistical Analysis Using Large Language Models.
TL;DR
While vibe coding has the potential to reduce barriers to data analysis for researchers, this case study demonstrated that it can produce both valid and invalid outputs and that foundational statistical training, knowledge, understanding, and methodological expertise remain paramount when using it.
Abstract
Background
Large language models have accelerated the adoption of generative artificial intelligence (AI), making AI tools more widely accessible through conversational prompting. One emerging application is vibe coding, in which users use natural-language prompts to generate code and desired outputs rather than manually writing traditional code.
Objective
To examine AI-assisted, human-in-the-loop (HITL) vibe coding as a proof of concept for data analysis, describe its components and a proposed workflow with explicit safeguards, and present a case study illustrating its use and potential failure points.
Methods
We used a proposed workflow that included framing research questions, operationalizing variables, organizing project folders, documenting decisions, applying retrieval-augmented generation, and using prompt engineering techniques. We used Cursor (v1.5.11) on a limited, clean admissions data set, in which admission status was modeled as a function of the Graduate Record Exam, grade point average, and undergraduate rank. Logistic regression was generated via conversational prompts, implemented in R, and the results were compared with a published reference output on a publicly available website.
Results
AI-assisted, HITL vibe coding produced statistical codes that included schema checks, range validations, data cleaning, exploratory analyses, regression modeling, and visualization. There were mixed results of both valid and invalid outputs. Regression coefficients, p values, and model fit statistics matched the outputs posted on the published reference output website. However, an error was identified in the predicted-probability confidence interval output, which was missed during the initial review of outputs.
Discussion
While vibe coding has the potential to reduce barriers to data analysis for researchers, this case study demonstrated that it can produce both valid and invalid outputs and that foundational statistical training, knowledge, understanding, and methodological expertise remain paramount when using it. Future studies should address important empirical questions about the use of vibe coding, such as under what conditions it can be safely used in research and what kinds of errors are most commonly generated when using it. AI-assisted HITL vibe coding should be used with caution and only with structured verification and safeguards, transparent reporting, and appropriate statistical and methodological oversight.