Skip to content

LLMs as Annotators: Evaluating Model–Human Alignment in Detecting Contentious Language in Historical Corpora

Ya-Hui Zhao Clemencia Siro L. Hollink OpenAI Josh Steven Achiam Sandhini Adler Lama Agarwal Ilge Ahmad Florencia Akkaya Diogo Aleman Janko Almeida Sam Altenschmidt Shyamal Alt-man A. Red Igor Avila Suchir Babuschkin Va-lerie Balaji Paul Balcom Haim-ing Baltescu Bao-Lin Mo Jeff Bavarian Ir-wan Belgum J. Bello Gabriel Berdine Christo-pher Bernadett-Shapiro Lenny Berner Oleg Bogdonoff Boiko Made Laine Anna-Luisa Boyd Greg Brakman Tim Brock-man Miles Brooks Kevin Brundage T. Button Rosie Cai Andrew Campbell Brittany Cann Chelsea Carey Rory Carlson B. Carmichael Che Chan Fotis Chang Derek Chantzis Sully Chen Ruby Chen C. Jason Mark Chen Benjamin L. Chen Chess Elie Georges C. Gibson Vik Goel Tarun Gogineni Gabriel Goh Raphael Gontijo-Lopes J. Gordon Morgan Grafstein Scott Gray Ryan Greene Shixiang Joshua Gross Shane Gu Yu-Fei Guo Chris Hallacy Jesse Han J. Harris Yu-Chen He Mike Heaton Johannes Heidecke Chris Hesse Alan Hickey W. Hickey P. Hoeschele Brandon Houghton Kenny Hsu Sheng-Li Hu Xin Hu Joost Huizinga Shantanu Jain Shawn Jain Joanne Jang Angela Jiang R. Jiang Hao-Zhun Jin Denny Jin Shino Jomoto Billie Jonn Hee-woo Jun T. Kaftan L. Kaiser A. Kamali I. Kanitscheider Nitish Shirish Tabarak Keskar K. Logan Jong Wook Kilpatrick Christina Kim Yongjik Kim Hendrik Kim J. Kirchner Matt Kiros Daniel Knight Łukasz Kokotajlo A. Kondraciuk Aris Kondrich Kyle Kon-stantinidis Gretchen Kosic V. Krueger Michael Kuo I. Lampe Teddy Lan Jan Lee J. Leike Daniel Leung C. Levy L. Rachel Molly Lim Stephanie L. Lin Mateusz Lin Theresa Teusz Litwin Ryan Lopez Patricia Lowe Lue Anna Kim Makanju Sam Malfacini Manning Todor Yaniv Markov B. Markovski Katie Mayer Andrew Mayne Bob McGrew S. McKinney Christine McLeavey Paul McMillan Jake McNeil David Medina Aalok Mehta Jacob Menick Luke Metz Andrey Mishchenko Pamela Mishkin Vinnie Monaco
· 0 citations · 29 references

TL;DR

It is suggested that LLMs can serve as auxiliary annotators for sensitive-language detection in historical materials, provided that human oversight and contextual interpretation remain central to annotation workflows.

View source

Similar papers

Open access Sep 2026

Using Large Language Models for Automated Corpus Annotation and Linguistic Analysis: A Critical Methodological Framework

This article argues against unvalidated full automation and proposes selective automation, disagreement-based human adjudication, model/version documentation, and preservation of raw outputs for corpus research and develops a validated human–LLM workflow for corpus research.

Maria Ibrar · 0 citations
#natural language process... Preprint Sep 2026

FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

FineWeb-CLaR is introduced, a large-scale annotated dataset derived from FineWeb and FineWeb-2 that places web documents on a shared culture-language-region axis for corpus auditing and benchmark alignment, enabling direct comparison between corpus-side pretraining evidence and benchmark-side evaluation coverage.

Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet et al. · 0 citations
Open access 2026

Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

A layer-wise analysis indicates that surface-level features such as temporality and negation are captured more reliably than deeper semantic phenomena like quantification in large language models, highlighting the limited capacity of current LLMs to generate fully formal meaning representations.

Rémi De Vergnette, Maxime Amblard · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.