High accuracy with low costs: the pretrain-finetune paradigm for classification with transformer-based language models

Abstract

Political science increasingly uses text classification to gauge subtle concepts such as toxicity or anger. Traditional methods treat words in isolation, overlooking the contextual dynamics where meaning resides. Transformer-based language models address these limitations but remain underutilized, especially among applied scholars, partly due to misconceptions about their mechanisms and computational requirements. This article bridges this gap by offering an accessible explication of the pretrain-finetune paradigm, focusing on underlying mechanisms and their potential to improve political text analysis by harnessing models trained on extensive datasets while requiring modest labeled data and computational resources. I demonstrate the approach by identifying toxic language in conversation threads following U.S. Senators’ tweets and provide an online tutorial to support nontechnical scholars in adopting the methodology.