ArCompose: Compositional Text to Image Synthesis for Modern Standard and Iraqi Arabic via Token Supervised Diffusion and NMT Driven Prompt Planning
TL;DR
By using an experimental with controlled single-dataset curriculum on COCO, the structural guidance and curriculum learning were proven to be more than adequate to overcome intricate dialectal and compositional ambiguities in generative AI.
Abstract
The latest diffusion-based text-to-image (T2I) models are capable of generating very realistic images, but they are easily destabilized by the compositional prompts involving multiple objects, attributes, and relations. Meanwhile, the use of multiple languages is on the increase, yet Arabic (particularly the dialectal ones) is underrepresented in the generative framework and in the evaluation metrics. In this paper, ArCompose is introduced, a compositional T2I system, which fills these two gaps through an integrative approach based on token-level supervision, a compositional curriculum, and a linguistically informed Arabic prompt planner. Beginning with Stable Diffusion version 1.5, auxiliary losses are added that both match noun tokens to pseudo segmentation masks at selected cross attention layers and enhance token to region grounding with no change in inference cost. A COCO-only three-stage curriculum is developed, which introduces the model to single-object, multi-object, and full complexity scenes gradually in order to stabilize training. An OPUS MT Arabic-English model is adapted on the language side, training it on the Modern Standard Arabic style and a small Iraqi dialect corpus, and uses it with spaCy dependency parsing to generate weighted, Compel-style English prompts on an Arabic input. The experiment on the T2I-CompBench demonstrates that ArCompose achieves a higher score (0.62) on the Attribute Binding (against 0.39 for the baseline) and 0.44 Relationship score (against 0.29). Moreover, using the ArCompose system on the VISOR benchmark, the object Accuracy is increased from 30.2% to 54.2%, which eliminates the difference between the system and state-of-the-art English-only systems such as TokenCompose. The NMT-based planner was more accurate, with BLEU scores on MSA captions of 44.6, and the token-monitored cost is almost zero at 3.3%. By using an experimental with controlled single-dataset curriculum on COCO, the structural guidance and curriculum learning (without massive pre-training on huge amounts of multilingual data) were proven to be more than adequate to overcome intricate dialectal and compositional ambiguities in generative AI.