Skip to content

Evaluating Style-Personalized Text Generation: Challenges and Directions

Aug 2025 · arXiv.org · Vol abs/2508.06374 · 2 citations · 79 references
Computer Science

TL;DR

This work critically examines the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges, and finds strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods.

Abstract

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.

View source

Similar papers

Preprint Aug 2026

When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era

AVShift is introduced, the first German benchmark for systematically evaluating AV under multiple distribution shifts, and feature analysis reveals stylistic features that remain stable across genres, while their relative importance varies depending on the specific genre transition.

Lotta Kiefer, Brisca Balthes, Christoph Leiter et al. · 0 citations
Preprint Aug 2026

PersonalBench: Measuring the Authorship Gap in LLM Personalization

PersonalBench is introduced, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics, which finds that personalization methods do produce author-differentiated output but this differen...

Yashwant Sawant · 1 citation

LLJ Cards: Best practices for the Use of LLMs as Judges

LLJ Cards is introduced, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations and provides a structured framework for applying these principles in the design and reporting of automated evalua...

Khaoula Chehbouni, Melina Medjdoub, Florian Carichon et al. · 0 citations

C-HAT-Bench: Benchmarking Chinese AI-Text Detection Beyond Fully Generated Text

A Chinese Human-AI Collaborative Text Detection Benchmark (C-HAT-Bench), a unified benchmark that links human-written source texts from five domains to variants produced using six generative models under Prefix-Conditioned Continuation as a reference setting and three collaborative production modes.

Qing Yang, Zi-Xiang Luo, Zhen-Yun Mao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

This work introduces a human-aligned evaluation framework for text-to-SVG generation, and develops two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcem...

Marco Cipriano, Leonardo Zini, Alexandra Schild et al. · 0 citations
#machine learning Preprint Sep 2026

Reason in Style: Discovering and Controlling Style in Language Models

An algorithm is designed that learns to separate representations of content and style from language models' outputs and validate its effectiveness on math questions in a controlled setting, finding that style affects correctness: the probability of solving a problem depends on the style the authors condition on, and di...

Ioana Marinescu, E. Oermann, Kyunghyun Cho · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.