Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

• URL: http://arxiv.org/abs/2603.12247v1
• Date: Thu, 12 Mar 2026 17:57:21 GMT
• Title: Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
• Abstract summary: Current reward models, which act as critics during reinforcement learning, often suffer from hallucinations and assign noisy scores.<n>We present FIRM, a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing.<n> FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models.
• Score: 67.26349227500084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.

Related papers

• RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing [9.858046310638803]
  We propose a pairwise generative reward modeling framework that produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images.<n>We train dedicatedRM models for text-to-image generation and image editing using a two-stage training pipeline: supervised fine-tuning teaches the model the model the rubric-based scoring, while further improves scoring through fine-level rewards.
  arXiv  Detail & Related papers  (2026-08-27T10:56:47Z)
• Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling [18.3502725927898]
  We introduce Edit- and Edit-Reward, a unified evaluation suite for image editing and reward modeling.<n> Edit-Reward includes 2,388 annotated instances spanning six progressively challenging task categories.<n>It adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed rubrics.
  arXiv  Detail & Related papers  (2026-05-13T06:33:54Z)
• ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning [86.61218827780675]
  We present ReasonEdit, an evaluation tool for text-guided image editing.<n>It is trained using reward signals derived from RE-Reward and the Group Relative Policy Optimization (GRPO) algorithm.<n>It is capable of generating high-quality interpretable evaluation text, enabling more transparent and trustworthy assessment for image editing.
  arXiv  Detail & Related papers  (2026-05-08T09:23:26Z)
• Leveraging Verifier-Based Reinforcement Learning in Image Editing [41.64251379221566]
  We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) for downstream image editing.<n>Our experiments demonstrate that our Edit-RRM surpasses powerful VLMs as an editing-specific reward model.
  arXiv  Detail & Related papers  (2026-04-30T06:54:39Z)
• Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content [71.46991494014382]
  We introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images.<n>Q-Real consists of 3,088 images generated by popular text-to-image models.<n>We construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning.
  arXiv  Detail & Related papers  (2025-11-21T02:43:17Z)
• EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling [71.8265422228785]
  Reinforcement Learning (RL) offers a promising solution, but its adoption in image editing has been hindered by the lack of a high-fidelity, efficient reward signal.<n>We present a comprehensive methodology to overcome this barrier, centered on the development of a state-of-the-art, specialized reward model.
  arXiv  Detail & Related papers  (2025-09-28T14:28:24Z)
• From Editor to Dense Geometry Estimator [77.21804448599009]
  We introduce textbfFE2E, a framework that adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction.<n>FE2E achieves over 35% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100$times$ data.
  arXiv  Detail & Related papers  (2025-09-04T15:58:50Z)
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step [86.69947123512836]
  Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks.<n>We provide the first comprehensive investigation of the potential of CoT reasoning to enhance autoregressive image generation.<n>We propose the Potential Assessment Reward Model (PARM) and PARM++, specialized for autoregressive image generation.
  arXiv  Detail & Related papers  (2025-01-23T18:59:43Z)
• CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling [27.86204841898399]
  Reward modeling in large language models is susceptible to reward hacking.<n>We propose Context-Aware Reward Modeling (CARMO) to mitigate this problem.<n>We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1% improvement on Reward Bench.
  arXiv  Detail & Related papers  (2024-10-28T21:18:49Z)
• RL for Consistency Models: Faster Reward Guided Text-to-Image Generation [15.238373471473645]
  We propose a framework for fine-tuning consistency models viaReinforcement Learning (RL) Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps.
  arXiv  Detail & Related papers  (2024-03-25T15:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.