Automated Text Summarization: A Comparative Study of Classical and Transformer Approaches
Abstract
One of the basic processes of Natural Language Processing (NLP) is text summarization where the goal is to be able to summarize large amounts of text and capture the gist and semantic structures while maintaining the factual accuracy. In the given paper, two extractive models (TextRank, BERTSUM) and three abstractive models that rely on transformers GPT-3.5, T5, PEGASUS, are analyzed in detail and compared. Through the lenses of automatic evaluation systems and human-centric evaluation systems, the models are tested using three well-known benchmark datasets that are CNN/Daily Mail, PubMed and XSum. The quantitative evaluation is based on commonly used metrics such as ROUGE-1, ROUGE-2, ROUGE-L and BLEU, while the qualitative evaluation involves a human raters' judgement of 100 summaries based on the four criteria of fluency, coherence, informativeness and conciseness. Experimental results show that the abstractive models always perform better than extractive models, while PEGASUS achieves the highest ROUGE-L score (0.45) that is 60% better than TextRank. By human ratings too, abstractive summaries outperform extractive ones, and they are more fluent and coherent (around 35 percent more) than the extractive ones, while extractive ones are more accurate with respect to the facts. However, this leads to reduced time costs, and abstractive models are 4-8 times slower than extractive models running and they take more hardware memory. The novelty of this work is in unitary comparative depth, the mixture of automated evaluation and human evaluation and systematic research of the efficiency of trade-offs of the quality of the summarization paradigms.