Message Over Modality: Toward a Source-Agnostic Message Evaluation Framework for AI-Delivered Social Support

Abstract

As artificial intelligence (AI) systems increasingly take on socially supportive roles, questions remain about how users perceive such support across different AI modalities (i.e., the form through which an AI system communicates, such as voice, embodiment, or text). This study examines early-stage evaluations formed after a single initial interaction with three common AI modalities, voice-based assistants (VBAs), social robots, and text-based generative AI, delivering either informational or emotional support. Using a 3 (AI modality: VBA vs. social robot vs. text-based generative AI) × 2 (support type: emotional vs informational) between-subjects experimental design, participants (N = 203) viewed scripted video vignettes of human–AI encounters and then evaluated the AI agents on credibility (competence, character, caring), task and social attraction, and perceived support. Results revealed that all agents were rated significantly above the midpoint on most attributes, suggesting users accept AI agents as credible and supportive. However, no significant differences emerged between AI types. Instead, support type had a stronger influence; informational support increased perceived competence, while emotional support elevated perceptions of caring. Qualitative responses supported these trends, emphasizing message relevance over agent form. These findings introduce and support the Source-Agnostic Message Evaluation (SAME) framework, which proposes that in early-stage, observational AI support contexts, users assess AI support based more on what is said than who, or what, says it. This has implications for AI design, emphasizing message quality over modality in user experience optimization.