Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

• URL: http://arxiv.org/abs/2607.11886v1
• Date: Mon, 13 Jul 2026 17:59:58 GMT
• Title: Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
• Abstract summary: SpectraReward turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning.<n>Self-SpectraReward forms a closed-loop self-improving framework without external reward models or external knowledge.
• Score: 52.00324244076857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

Related papers

• Leveraging Verifier-Based Reinforcement Learning in Image Editing [41.64251379221566]
  We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) for downstream image editing.<n>Our experiments demonstrate that our Edit-RRM surpasses powerful VLMs as an editing-specific reward model.
  arXiv  Detail & Related papers  (2026-04-30T06:54:39Z)
• RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward [64.78078130943489]
  We introduce RetouchIQ, a framework that performs instruction-based executable image editing through MLLM agents guided by a reward model.<n>We show that RetouchIQ substantially improves both semantic consistency and perceptual quality over previous MLLM-based and diffusion-based editing systems.
  arXiv  Detail & Related papers  (2026-02-19T17:11:59Z)
• The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation [52.648073272395635]
  We introduce Adv-GRPO, an RL framework with an adversarial reward that iteratively updates both the reward model and the generator.<n>Unlike KL regularization that constrains parameter updates, our learned reward directly guides the generator through its visual outputs.<n>In human evaluation, our method outperforms Flow-GRPO and SD3, achieving 70.0% and 72.4% win rates in image quality and aesthetics, respectively.
  arXiv  Detail & Related papers  (2025-11-25T12:35:57Z)
• FreeRet: MLLMs as Training-Free Retrievers [21.04237443940747]
  FreeRet is a plug-and-play framework that turns any MLLM into a two-stage retriever.<n>On the MMEB and MMEB-V2 benchmarks, FreeRet substantially outperforms models trained on millions of pairs.
  arXiv  Detail & Related papers  (2025-09-29T11:28:42Z)
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback [33.127607245587576]
  We introduce a framework that enables MLLMs to learn complex visual reasoning from only raw images.<n>We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning.<n>The RRVF-trained model not only outperforms existing MLLMs and supervised fine-tuning baselines but also exhibits superior generalization.
  arXiv  Detail & Related papers  (2025-07-28T12:21:19Z)
• Generalist Reward Models: Found Inside Large Language Models [50.7432354447554]
  We show that a powerful reward model is already latently present within any Large Language Models (LLMs) trained via standard next-token prediction.<n>We prove that this endogenous reward is not a reward function learned through offline inverse reinforcement learning.<n>We also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model.
  arXiv  Detail & Related papers  (2025-06-29T13:45:54Z)
• Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems [54.4392552373835]
  Reward models (RMs) are crucial for the training and inference-time scaling up of large language models (LLMs)<n>We propose agentic reward modeling, a reward system that combines reward models with verifiable correctness signals to provide reliable rewards.<n>We conduct comprehensive experiments on existing reward model benchmarks and inference time best-of-n searches on real-world downstream tasks.
  arXiv  Detail & Related papers  (2025-02-26T17:19:12Z)
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback [24.759613248409167]
  Reward engineering has long been a challenge in Reinforcement Learning research. We propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains.
  arXiv  Detail & Related papers  (2024-02-06T04:06:06Z)
• Rethinking the Role of Proxy Rewards in Language Model Alignment [39.53237479058083]
  We study the role of proxy rewards in the Large Language Models alignment via reverse reward engineering' We aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals. Our findings indicate that successfully emulating the gold reward requires generating responses that are relevant with enough length to open-ended questions.
  arXiv  Detail & Related papers  (2024-02-02T11:58:08Z)
• Language Reward Modulation for Pretraining Reinforcement Learning [61.76572261146311]
  We propose leveraging the capabilities of LRFs as a pretraining signal for reinforcement learning. Our VLM pretraining approach, which is a departure from previous attempts to use LRFs, can warmstart sample-efficient learning on robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-08-23T17:37:51Z)
• DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models [97.31200133440308]
  We propose using online reinforcement learning to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an RL problem. Our approach, coined DPOK, integrates policy optimization with KL regularization.
  arXiv  Detail & Related papers  (2023-05-25T17:35:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.