Aug 2025· arXiv.org· Vol abs/2508.06374· 2 citations· 79 references
Computer Science
TL;DR
This work critically examines the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges, and finds strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods.
Abstract
With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.
AVShift is introduced, the first German benchmark for systematically evaluating AV under multiple distribution shifts, and feature analysis reveals stylistic features that remain stable across genres, while their relative importance varies depending on the specific genre transition.
Lotta Kiefer, Brisca Balthes, Christoph Leiter et al.· 0 citations
PersonalBench is introduced, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics, which finds that personalization methods do produce author-differentiated output but this differen...
LLJ Cards is introduced, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations and provides a structured framework for applying these principles in the design and reporting of automated evalua...
A Chinese Human-AI Collaborative Text Detection Benchmark (C-HAT-Bench), a unified benchmark that links human-written source texts from five domains to variants produced using six generative models under Prefix-Conditioned Continuation as a reference setting and three collaborative production modes.
Qing Yang, Zi-Xiang Luo, Zhen-Yun Mao et al.· arXiv.org· 0 citations
This work introduces a human-aligned evaluation framework for text-to-SVG generation, and develops two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcem...
Marco Cipriano, Leonardo Zini, Alexandra Schild et al.· 0 citations
An algorithm is designed that learns to separate representations of content and style from language models' outputs and validate its effectiveness on math questions in a controlled setting, finding that style affects correctness: the probability of solving a problem depends on the style the authors condition on, and di...
Ioana Marinescu, E. Oermann, Kyunghyun Cho· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.