Don’t Believe the Hype: Methodological Approaches for Applying LLM-Assisted Content Analysis to Reported Speech in Journalism

Abstract

To better understand how journalists represent sources, it is necessary to systematically study the use of reported speech in news coverage. This paper presents a method for LLM-assisted content analysis to identify and classify reported speech in Dutch newspapers automatically. The study evaluates a three-step procedure utilising role-based instructions to prompt the model as a professional journalist. First, a codebook for identifying citation structures and source types was developed with LLM support and then manually verified. Second, inter-coder reliability between human coders and the LLM was assessed on a representative sample of Dutch news articles using a human-in-the-loop validation approach. Third, the prompt-engineered LLM was used to code a large corpus spanning seven decades (1950–2024). Manual verification of 16,689 citations shows a weighted F1-score of 0.75, which aligns with recent benchmarks for high-capacity models performing complex journalistic coding. While human oversight remains the benchmark for reliability, due to issues such as repeated citations that were given as examples in the used prompts and representational bias, LLM-based systems perform sufficiently well for large-scale analyses of journalistic source use. The paper concludes that hybrid human–AI workflows provide a practical bridge between traditional rule-based approaches and new generative models, offering scalable and cost-effective methods for studying source representation in journalism.