Reading Between the Lines: Evaluating GenAI's Detection of Attitudinal Meaning in News Discourse
Abstract
This study investigates the potential and limitations of ChatGPT in identifying attitudinal resources in news discourse. Fifteen news articles were analyzed by ChatGPT, and the results generated by it were compared with manual annotations obtained in a previous study. The findings show that ChatGPT is capable of identifying attitudinal resources and generally captures the overall evaluative orientation of the texts. Compared with human annotators, however, ChatGPT tends to identify fewer positive resources but more neutral ones. The analysis also reveals several limitations of ChatGPT in attitude identification. It tends to over mark a particular type of attitudinal resources within a single discourse; moreover, its responses are not always stable, as the identification and interpretation process may be continuously revised when it is challenged through subsequent prompts. These findings suggest that, although ChatGPT has considerable potential as an auxiliary tool for discourse analysis and may improve the efficiency of large-scale attitudinal analysis, its analytical results require careful human validation. At present, it should be regarded only as a complementary tool rather than a substitute for human in discourse research.