AGTI-Bench: A Human-Aligned Benchmark for Text-Aware Text-to-Image Generation

Abstract

Rendering clear and accurate text is a critical capability of text-to-image (T2I) models. Despite rapid advances in T2I generation, the systematic and human-aligned evaluation of text rendering remains underexplored. Existing benchmarks often suffer from limited scenario coverage, a lack of bilingual support, and an over-reliance on unvalidated automated metrics. To address this gap, we introduce AGTI-Bench, a dedicated benchmark for evaluating text-aware AI-generated text images (AGTIs). AGTI-Bench is characterized by three core features: (1) Bilingual and Diverse Scenarios, covering typical real-world text-bearing image types with native Chinese--English support; (2) Multi-dimensional Evaluation, focusing on text accuracy, text rendering quality, and overall image quality; and (3) Human Preference Alignment, supported by large-scale subjective annotations. Our analysis reveals that while OCR-based methods excel at verifying text content, multimodal large language model (MLLM) evaluators achieve stronger alignment with human preferences for perceptual quality assessment. Furthermore, our benchmarking of representative T2I models uncovers significant performance disparities across the three evaluation dimensions, underscoring the necessity of multi-dimensional assessment for text-aware image generation. The dataset and code are publicly available at https://github.com/mzcha0/AGTI-Bench-dataset.