Evaluation of Inductive Coding with LLMs

Abstract

Large Language Models (LLMs) like ChatGPT are reshaping qualitative research by offering data-driven analysis. Recent studies focus on labeling and classifying qualitative data, yet the generative process behind code system development is underexplored. This study investigates how ChatGPT constructs coding systems from interview data under iterative code engineering. The resulting code system is systematically compared to an inductive coding framework derived by content analysis—quantitatively by SBERT analysis, Jaccard index, and network analysis, and qualitatively. The results of NLP-based analysis show that there are mostly high cosine similarities in the one-to-one mapping between ChatGPT codes and content analysis. The overall Jaccard index is low, indicating limited overlap between the coding units and, consequently, that the two coding systems often did not refer to the same content. However, the mapped codes showed substantial variation: while nearly half exhibited high overlap, others showed no overlap. Network analysis shows that network density is comparable across the two systems, indicating a similar level of interconnectedness among codes despite differences in network size. Finally, iterative prompt engineering could create a stable code system, but ChatGPT lacks in domain-specific coding addressing the research questions adequately, warranting further investigation across larger datasets.