Long-form RewardBench: Evaluating Reward Models for Long-form Generation

• URL: http://arxiv.org/abs/2603.12963v1
• Date: Fri, 13 Mar 2026 13:05:17 GMT
• Title: Long-form RewardBench: Evaluating Reward Models for Long-form Generation
• Abstract summary: Long-form RewardBench is the first reward modeling testbed specifically designed for long-form generation.<n>Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning.<n>Our findings reveal that current models still lack long-form reward modeling capabilities.
• Score: 61.60385107031075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread adoption of reinforcement learning-based alignment highlights the growing importance of reward models. Various benchmarks have been built to evaluate reward models in various domains and scenarios. However, a significant gap remains in assessing reward models for long-form generation, despite its critical role in real-world applications. To bridge this, we introduce Long-form RewardBench, the first reward modeling testbed specifically designed for long-form generation. Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning. We collected instruction and preference data through a meticulously designed multi-stage data collection process, and conducted extensive experiments on 20+ mainstream reward models, including both classifiers and generative models. Our findings reveal that current models still lack long-form reward modeling capabilities. Furthermore, we designed a novel Long-form Needle-in-a-Haystack Test, which revealed a correlation between reward modeling performance and the error's position within a response, as well as the overall response length, with distinct characteristics observed between classification and generative models. Finally, we demonstrate that classifiers exhibit better generalizability compared to generative models trained on the same data. As the first benchmark for long-form reward modeling, this work aims to offer a robust platform for visualizing progress in this crucial area.

Related papers

• Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewards [40.23960862004138]
  This paper investigates whether state-of-the-art small language models can be turned into usable reward models.<n>We construct a dataset of code samples with correctness labels derived from the APPS coding challenge benchmark.<n>Using this critic, we achieve over a 20% improvement in the search capability of the most accurate code out of multiple generations.
  arXiv  Detail & Related papers  (2025-10-27T07:36:41Z)
• GRAM: A Generative Foundation Reward Model for Reward Generalization [48.63394690265176]
  We develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning.<n>This model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning.
  arXiv  Detail & Related papers  (2025-06-17T04:34:27Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• Exploring Training and Inference Scaling Laws in Generative Retrieval [50.82554729023865]
  Generative retrieval reformulates retrieval as an autoregressive generation task, where large language models generate target documents directly from a query.<n>We systematically investigate training and inference scaling laws in generative retrieval, exploring how model size, training data scale, and inference-time compute jointly influence performance.
  arXiv  Detail & Related papers  (2025-03-24T17:59:03Z)
• PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis [8.785345412061792]
  We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations.<n>Our approach includes a modular and user-friendly model development and evaluation platform.<n>We highlight the limitations of widely used models, such as mode collapse.
  arXiv  Detail & Related papers  (2024-08-20T07:40:20Z)
• Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development [67.55944651679864]
  We present a new sandbox suite tailored for integrated data-model co-development.<n>This sandbox provides a feedback-driven experimental platform, enabling cost-effective and guided refinement of both data and models.
  arXiv  Detail & Related papers  (2024-07-16T14:40:07Z)
• DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging [65.41765072566287]
  We propose textbfDomain knowledtextbfge merged textbfReward textbfModel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging.
  arXiv  Detail & Related papers  (2024-07-01T17:01:54Z)
• A Two-Phase Recall-and-Select Framework for Fast Model Selection [13.385915962994806]
  We propose a two-phase (coarse-recall and fine-selection) model selection framework. It aims to enhance the efficiency of selecting a robust model by leveraging the models' training performances on benchmark datasets. It has been demonstrated that the proposed methodology facilitates the selection of a high-performing model at a rate about 3x times faster than conventional baseline methods.
  arXiv  Detail & Related papers  (2024-03-28T14:44:44Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.