Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Runhu Huang Qihui Zhang Zhe Liu Yu Gao Jie Wu Hengshuang Zhao

Jul 2026 · arXiv.org · Vol abs/2607.11886 · 0 citations · 63 references

Computer Science

TL;DR

SpectraReward is proposed, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning, and Self-SpectraReward is introduced, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch.

Abstract

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

This work proposes REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage RL-distillation co-training framework that attaches a decoupled student to an arbitrary RL teacher that enables few-step CFG-free inference that matches or surpasses its 40-step RL teacher, with an overall additional training cost below 25% over pure RL.

Yuhan Li, Fan-Gao Zeng, Sicong Kang et al. · 0 citations

Preprint Aug 2026

HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models

This work proposes Hybrid-Policy Self-Distillation (HPSD), a novel self-distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced prompt, while the student runs in the base T2V mode with only the vanilla prompt.

Jiazi Bu, Pengyang Ling, Yujie Zhou et al. · 2 citations

#reinforcement learning Preprint Sep 2026

Test-Time Weak-to-Strong Alignment: Transferring Implicit Rewards from Weak to Strong Flow Models

The method, AlignGraft, aligns a larger, frozen, never-tuned model by adding the pair's velocity difference during sampling by adding the pair's velocity difference during sampling, and preserves the large model's fidelity at a small constant sampling overhead.

Xin Xie, Fan Zhang, Dong Gong · 0 citations

#machine learning Preprint Sep 2026

Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning

Test-time reinforcement learning can adapt vision-language models (VLMs) to unlabeled target data, but its effectiveness is fundamentally limited by the reliability of self-generated learning signals. To assess the reliability of consensus-based learning signals, we analyze VLM test-time reinforcement learning across diverse VQA datasets and model sizes, revealing two limitations. First, gains from consensus-based test-time training largely come from answer normalization rather than content correction. Second, many initial VLM responses are incorrect due to the model's limited ability to jointly use the image and the question; consensus rewards derived from these outputs may preserve the resulting grounding errors rather than correct them. Motivated by these, we propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation. TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants, using the resulting token-level likelihood changes to estimate dependence on each input. It combines image and question dependence with calibrated confidence to construct a response-level reward that favors jointly grounded responses, and uses the token-level signals to assign greater positive policy credit to tokens supported by both inputs. This design favors responses that are jointly grounded in the image and the question and sufficiently confident, rather than merely popular. Experiments across eight VQA datasets and multiple VLM sizes show that TTIQ achieves the best average performance at every model scale. It further generalizes across VLM families, while models trained on one dataset improve performance on unseen datasets without further training.

Xinrui He, Ting-Wei Li, Jun-Ting Wang et al. · 0 citations

Preprint Aug 2026

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

The sample efficiency and scalability of RL post-training for video MLLMs and introduces OraRL, a decoupled advantage estimator that scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts.

Yunheng Li, Guohong Mu, Hao Li et al. · 0 citations

Jul 2026

Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation

This work proposes NOPD (Noisy Student On-Policy Self-Distillation), a simple yet effective self-distillation approach that improves VLMs without any external models or ground-truth answers, and demonstrates that NOPD is a general approach to enhance VLMs, achieving improvements across three models on 12 benchmarks.

Shuai Wang, Daoan Zhang, Zhe Tang et al. · arXiv.org · 0 citations