Evaluating Summaries of Social Media Posts: Balancing Fairness for Readers and Authors

Abstract

Text summarization has a wide range of applications, from condensing lengthy documents to summarizing user-generated content (such as tweets, Facebook posts, and Reddit discussions) that reflect diverse viewpoints on socially significant topics. Traditionally, summarization has been considered a reader-centric task, and algorithms are evaluated against human-written reference summaries using metrics such as the ROUGE score. However, this approach has two primary limitations when applied to user-generated content, especially written on sensitive issues. First, through surveys with participants from various countries, we demonstrate that human-written reference summaries can contain implicit biases. These biases introduce significant variations in the evaluation of summarization algorithms, raising concerns about their reliability in identifying summaries that effectively meet reader expectations. Second, we show that algorithmically generated summaries, as ranked by traditional evaluation methods, often fail to align with authors’ preferences. These issues render the existing summary evaluation framework suboptimal for both readers and the authors of the content being summarized. In this work, by reimagining summarization of user-generated content as a two-sided problem, we emphasize the importance of satisfying both readers and authors. Specifically, we take a two-pronged approach to tackle the issues with existing evaluation setup: (i) we introduce the concept of inequality in reader satisfaction to tackle the implicit biases in reference summaries; and (ii) we propose a novel evaluation framework based on satisfying the authors of the content. More specifically, we present an author satisfaction-based evaluation metric CROSSEM which, we show empirically, can complement the current summary evaluation paradigm. This work represents the first attempt to develop a fair summary evaluation framework for social media posts and is likely to spark future research in this domain.