Who Do We Trust? Investigating the Human-AI Trust Gap in Scientific Social Media Discourse

Abstract

As social media continues to be one of the primary communication channels, it has become a critical hub for the dissemination of scientific information. The nature of these platforms allows for the quick sharing of scientific claims, even without a reference. Users often rely on contextual cues such as comments accompanying posts for validation, while some platforms are increasingly integrate LLM-based systems that participant in online discourse by generating comments, explanations, and fact-checks. In this paper, we investigate how comment attribution (Human vs. AI) affects perceived trustworthiness of comments and posts and how it changes with the presence of a reference to a scientific publication and the comment's stance. We conducted a preregistered within-subjects user study (N=152) using a web-based platform that simulates a social media timeline containing posts and comments. We compared eight conditions: two types of comment attribution (Human vs. AI), two types of comment stance (Endorsing vs. Refuting), and two types of reference existence (Reference vs. No Reference). While we did not find evidence for a difference in overall trustworthiness between Human- and AI- attributed comments, we found that comments without a reference that were attributed to an AI were perceived as more trustworthy than those attributed to a human, highlighting the risk of over-reliance on AI. Exploratory analysis further indicate differences in comment and post trustworthiness between Human- and AI-attributed comments, depending on whether posts' claims aligned with scientific consensus. These observed differences suggest a potential interplay of trust and confirmation bias in the context of AI-attributed comments, which warrants further investigation by future research.