SON: Enhancing Prompt Understanding of Diffusion Models with Large Language Models Guided Layouts

Simile Understanding in Text-to-Image Models: An Evaluation Framework

Similes provide a compact and expressive way to describe visual characteristics in text prompts. Recent text-to-image models (t2i models) can produce visually compelling outputs from simile prompts, yet even frontier models frequently misinterpret the metaphorical vehicle and confuse it with the object. These systematic failures reveal a gap between figurative language and object-level visual grounding in t2i models. To investigate this issue, we propose a scalable evaluation framework for simile understanding. Our framework includes (1) a controlled simile dataset in which metaphorical vehicles are drawn from a predefined set of object-detectable categories and combined with diverse templates, (2) automatic grounding metrics based on YOLO (You Only Look Once) detection, and (3) text encoder layer analysis using Diffusion Lens to track how metaphorical vehicles emerge during generation. Experiments across architecturally diverse t2i models reveal consistent literalization failure patterns. We further discuss potential mitigation strategies for improving simile grounding in t2i models.

Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis.

The evolution of large models has witnessed the emergence of In-Context Learning (ICL) capabilities. In Natural Language Processing (NLP), numerous studies have demonstrated the effectiveness of ICL. Inspired by the success of Large Language Models (LLMs), researchers have developed Large Multimodal Models (LMMs) with ICL capabilities. However, explorations of demonstration configuration for multimodal ICL remain preliminary. Additionally, the controllability of In-Context Examples (ICEs) provides an efficient and cost-effective means to observe and analyze the inference characteristics of LMMs under varying inputs. This paper conducts a comprehensive external and internal investigation of multimodal in-context learning on the image captioning task. Externally, we explore demonstration configuration strategies through three dimensions: shot number, image retrieval, and caption assignment. We employ multiple metrics to systematically and thoroughly evaluate and summarize key findings. Internally, we analyze typical LMM attention characteristics and develop attention-based metrics to quantify model behaviors. We also conduct auxiliary experiments to explore the feasibility of attention-driven model acceleration and compression. We further compare performance variations between LMMs with identical model design and pretraining strategies and explain the differences from the angles of pre-training data features. Our study reveals both how ICEs configuration strategies impact model performance through external experiments and characteristic typical patterns through internal inspection, providing dual perspectives for understanding multimodal ICL in LMMs. Our method of combining external and internal analysis to investigate large models, along with our newly proposed metrics, can be applied to broader research areas.

Prompt Yourself: Awakening Textual Semantics in 1D Visual Tokenizers

VQA-Guided Diffusion: Enhancing Text-to-Image Generation With Semantic Feedback From Visual Question Answering

Diffusion models for text-to-image (T2I) synthesis, e.g. Stable Diffusion, generate visually realistic images, but often fail to capture the fine-grained semantic nuances of complex prompts. Typical failure modes include missing objects, incorrect attribute assignment e.g. wrong colors, distorted spatial relationships and inaccurate object counts. This paper proposes a VQA-Guided Text-to-Image Feedback Framework to address these limitations, substituting the traditional one-shot generation with an iterative self-correcting loop. In our framework, an input prompt is first decomposed into its semantic components (objects, properties, relationships) using natural language processing. Then a set of Visual Question Answering (VQA) questions is automatically generated stepwise to probe object existence, colors, spatial relations and quantities. A VQA model (BLIP) answers the questions on the initially generated image from Stable Diffusion. Whenever it detects a semantic mismatch, it automatically refines the prompt (for example, by adding explicit corrective instructions such as “ make sure the car is red ”), and feeds the revised prompt back into the diffusion model. This cycle is repeated until adequate semantic alignment is obtained, with the framework incorporating various assessment criteria (CLIP Score, VQA Accuracy, LPIPS, picture quality score, and consistency loss) integrated into a unified multi-metric evaluation framework. For our experiments on MS-COCO captions, DrawBench, and a custom set of challenging prompts, the proposed method resulted in the following improvements observed on our test sets: the average CLIP Score improved from 0.311 (baseline Stable Diffusion) to 0.364, a relative increase of 17%; VQA accuracy was above 87% for all prompts evaluated; and in a human preference study of 39 pairwise comparisons, the VQA-guided method was preferred in 76.9% of the cases. Ablation experiments demonstrated the beneficial contribution of many components of the framework. We show that VQA can serve as an effective semantic feedback to significantly enhance prompt-image alignment without retraining the diffusion model, providing a powerful, interpretable and self-correcting strategy for text-to-image production.

DynEval: Holistic Evaluations of T2I Generative Models in the Wild

DynEval, a Dynamic Evaluation framework designed to jointly assess text-to-image alignment and image quality of T2I models, is introduced and a compact evaluator achieves a higher overall correlation with human judgments.

Towards Hierarchical Structure Understanding of Newspaper Images

Understanding newspaper images remains a challenging task due to their complex, nested hierarchical structures and dense, heterogeneous layouts. In this paper, we explore two complementary approaches for newspaper structure understanding. First, we present a modular bottom-up pipeline that combines state-of-the-art open-source models: YOLO for layout detection, LayoutReader for reading order prediction, and a custom algorithm for article segmentation. This approach leverages existing robust components while maintaining flexibility and interpretability. Second, we introduce Tiramisu (Tiered Transformers for Hierarchical Structure Understanding), a novel end-to-end transformer-based architecture that explicitly models document hierarchy through an iterative tiered process. Tiramisu performs section and article separation, block localization, semantic categorization, and reading order prediction using highly parallelized attention mechanisms. Finally, we release Finlam La Libert\'e, a new dataset designed specifically for evaluating hierarchical information retrieval in historical newspapers. Experimental results demonstrate the effectiveness of both approaches in reconstructing complex newspaper hierarchies, with comparative analysis highlighting their respective strengths for scalable document digitization. The Tiramisu training code, including the synthetic newspaper generator, is available at https://git.litislab.fr/tiramisu/tiramisu-newspaper-articles-extractor.