EQA-RM: A Generative Embodied Reward Model with Test-time Scaling

• URL: http://arxiv.org/abs/2506.10389v1
• Date: Thu, 12 Jun 2025 06:25:04 GMT
• Title: EQA-RM: A Generative Embodied Reward Model with Test-time Scaling
• Authors: Yuhang Chen, Zhen Tan, Tianlong Chen,
• Abstract summary: We introduce EQA-RM, a novel generative multimodal reward model specifically architected for Embodied Question Answering (EQA)<n>The generative nature of EQA-RM provides interpretable, structured reward feedback (beyond simple scalars)<n>We introduce EQARewardBench, a new benchmark built on OpenEQA for standardized EQA reward model assessment.
• Score: 43.79542068802909
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward Models (RMs), vital for large model alignment, are underexplored for complex embodied tasks like Embodied Question Answering (EQA) where nuanced evaluation of agents' spatial, temporal, and logical understanding is critical yet not considered by generic approaches. We introduce EQA-RM, a novel generative multimodal reward model specifically architected for EQA, trained via our innovative Contrastive Group Relative Policy Optimization (C-GRPO) strategy to learn fine-grained behavioral distinctions. The generative nature of EQA-RM provides interpretable, structured reward feedback (beyond simple scalars), uniquely enabling test-time scaling to dynamically adjust evaluation granularity, from concise scores to detailed critiques of reasoning and grounding, at inference without retraining. Concurrently, we introduce EQARewardBench, a new benchmark built on OpenEQA for standardized EQA reward model assessment. Demonstrating high sample efficiency, EQA-RM (fine-tuning Qwen2-VL-2B-Instruct) achieves 61.9\% accuracy on EQA-RM-Bench with only 700 samples, outperforming strong proprietary baselines, including Gemini-2.5-Flash, GPT-4o, Claude-3.5-Haiku, and open-sourced state-of-the-art models such as RoVRM and VisualPRM. The code and dataset can be found here https://github.com/UNITES-Lab/EQA-RM.

Related papers

• Discriminative Policy Optimization for Token-Level Reward Models [55.98642069903191]
  Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs)<n>Q-RM explicitly learns token-level Q-functions from preference data without relying on fine-grained annotations.<n>Reinforcement learning with Q-RM significantly enhances training efficiency, achieving convergence 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH.
  arXiv  Detail & Related papers  (2025-05-29T11:40:34Z)
• RM-R1: Reward Modeling as Reasoning [81.50471199906738]
  Reasoning Reward Models (ReasRMs) formulate reward modeling as a reasoning task.<n>We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1.<n>Our models achieve state-of-the-art performance across three reward model benchmarks on average.
  arXiv  Detail & Related papers  (2025-05-05T06:11:12Z)
• Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets [9.785129730843435]
  We introduce Dynamic-KGQA, a scalable framework for generating adaptive QA datasets from knowledge graphs.<n>Unlike fixed benchmarks, Dynamic-KGQA generates a new dataset variant on every run while preserving the underlying distribution.<n>Dynamic-KGQA produces compact, semantically coherent subgraphs that facilitate both training and evaluation of KGQA models.
  arXiv  Detail & Related papers  (2025-03-06T23:58:01Z)
• Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology [0.34530027457862006]
  Previous research has shown that existing models, when trained on EQA datasets that include unanswerable questions, demonstrate a significant lack of robustness. Our proposed training method includes a novel loss function for the EQA problem and challenges an implicit assumption present in numerous EQA datasets. Our models exhibit significantly enhanced robustness against two types of adversarial attacks, with a performance decrease of only about a third compared to the default models.
  arXiv  Detail & Related papers  (2024-09-29T20:35:57Z)
• Boosting CLIP Adaptation for Image Quality Assessment via Meta-Prompt Learning and Gradient Regularization [55.09893295671917]
  This paper introduces a novel Gradient-Regulated Meta-Prompt IQA Framework (GRMP-IQA) The GRMP-IQA comprises two key modules: Meta-Prompt Pre-training Module and Quality-Aware Gradient Regularization. Experiments on five standard BIQA datasets demonstrate the superior performance to the state-of-the-art BIQA methods under limited data setting.
  arXiv  Detail & Related papers  (2024-09-09T07:26:21Z)
• Learning Answer Generation using Supervision from Automatic Question Answering Evaluators [98.9267570170737]
  We propose a novel training paradigm for GenQA using supervision from automatic QA evaluation models (GAVA) We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art.
  arXiv  Detail & Related papers  (2023-05-24T16:57:04Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Degraded Reference Image Quality Assessment [23.871178105179883]
  We make one of the first attempts to establish a new paradigm named degraded-reference IQA (DR IQA) Specifically, we lay out the architectures of DR IQA and introduce a 6-bit code to denote the choices of configurations. We construct the first large-scale databases dedicated to DR IQA and will make them publicly available.
  arXiv  Detail & Related papers  (2021-10-28T05:50:59Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.