The Role of Language Understanding in Speech-Based Multimodal Automatic Personality Perception

Abstract

This work investigates the interplay between language (what people say) and paralanguage (how they say it) in speech-based Automatic Personality Perception, the task of predicting the personality traits that listeners attribute to speakers. The experiments yield two main insights. First, the language modality enables effective personality prediction regardless of whether listeners understand the speakers’ linguistic content. Second, a multimodal approach outperforms both individual modalities only when listeners understand what the speakers say, at least when the task is framed as a binary classification. These findings are important because they clarify the relative contributions of language and paralanguage to Automatic Personality Perception. The experiments involved 322 speakers and 200 personality raters, 100 of whom understood the speakers’ language and 100 of whom did not. Both the data and the code used in the experiments are publicly available in the repository referenced in this paper.