Evaluating LLM-Based Topic Interpretation and Sentiment Classification Against LDA-Human and Human-Coded Benchmarks in Consumer Review Analytics
TL;DR
This empirical study compares performance of ChatGPT, Claude AI, and BARD to model topics on 547 Amazon reviews of 2 water filter brands and shows that LDA produced more coherent, detailed themes, while LLM-based topic modeling generated broader, more semantically diffused clusters, highlighting a trade-off between interpretability and abstraction.
Abstract
Customer reviews provide valuable insights, but the large volume of review volumes makes it difficult to synthesize manually. AI-based language processing methods, including Latent Dirichlet Allocation (LDA) topic modeling, sentiment analysis, and large language models (LLMs), offer efficient automated analysis, but their reliability remains underexamined. This empirical study compares performance of ChatGPT, Claude AI, and BARD to model topics on 547 Amazon reviews of 2 water filter brands. Their outputs were compared with LDA topic modeling, supplemented by human evaluation. Sentiment analysis reliability was assessed by comparing Azure, ChatGPT, Claude AI, and BARD outputs with human judgments for 100 reviews. Results showed that LDA produced more coherent, detailed themes, while LLM-based topic modeling generated broader, more semantically diffused clusters, highlighting a trade-off between interpretability and abstraction. ChatGPT and Claude AI showed stronger alignment with human sentiment judgments, while Azure was more conservative and often produced more neutral or mixed classifications.