DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

• URL: http://arxiv.org/abs/2605.09269v1
• Date: Sun, 10 May 2026 02:32:19 GMT
• Title: DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
• Abstract summary: We introduce DeltaRubric, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM.<n>DeltaRubric operates in two steps: acting first as a $textitDisagreement Planner$, it generates a neutral, instance-specific verification checklist.<n>We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities.
• Score: 38.65432183778467
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce $\textbf{DeltaRubric}$, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a $\textit{Disagreement Planner}$, the model generates a neutral, instance-specific verification checklist. Transitioning into a $\textit{Checklist Verifier}$, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by $\textbf{+22.6}$ (4B) and $\textbf{+18.8}$ (8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

Related papers

• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• Beyond Output Critique: Self-Correction via Task Distillation [36.44752912823049]
  We propose a framework that introduces an intermediate step of task abstraction before solution refinement.<n>Given an input and an initial response, the model first distills the task into a structured template that captures key variables, constraints, and problem structure.<n>This abstraction then guides solution instantiation, grounding subsequent responses in a clearer understanding of the task.
  arXiv  Detail & Related papers  (2026-01-31T19:15:41Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards [13.663839318595505]
  We seek to answer what it would take to simultaneously align a model across various domains spanning those with verifiable and non-verifiable rewards.<n>We propose a unified framework that standardizes process reward model (PRM) training across both verifiable and non-verifiable settings.<n> Experiments across math reasoning, value alignment, and multi-turn dialogue show that our framework improves performance across multiple objectives simultaneously.
  arXiv  Detail & Related papers  (2025-10-01T17:54:15Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models [4.875712300661656]
  We present SCORE ($mathbfS$ystematic $mathbfCO$nsistency and $mathbfR$obustness $mathbfE$valuation), a comprehensive framework for non-adversarial evaluation of Large Language Models.<n>The SCORE framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency.
  arXiv  Detail & Related papers  (2025-02-28T19:27:29Z)
• OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning [15.59540726867483]
  We argue that in guided decoding, assessing the potential of an incomplete reasoning path can be more advantageous than simply ensuring per-step correctness. Inspired by the findings that $textitoutcome supervision for guided decoding essentially acts as a value model, we propose Outcome-supervised Value Model (OVM) Our experiments on two multi-step mathematical reasoning datasets, GSM8K and Game of 24, demonstrate the superior performance of the OVM model.
  arXiv  Detail & Related papers  (2023-11-16T09:56:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.