Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models

• URL: http://arxiv.org/abs/2603.20212v1
• Date: Mon, 02 Mar 2026 15:48:54 GMT
• Title: Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models
• Abstract summary: We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory.<n>It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking)<n>F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%.
• Score: 16.460841602259787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%. Code and data will be publicly available.

Related papers

• Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling [0.68233044479196]
  Reward models in RLHF are trained to score only the final token of a response.<n>We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward.<n>We introduce Temporally Coherent Reward Modeling, which induces this property via two regularization terms on top of the standard Bradley-Terry loss.
  arXiv  Detail & Related papers  (2026-04-24T19:49:56Z)
• Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty [33.04999074297977]
  E-GRM is an efficient generative reward modeling framework grounded in model-internal uncertainty.<n>We show that E-GRM substantially reduces inference cost while consistently improving answer accuracy.
  arXiv  Detail & Related papers  (2026-04-11T07:35:08Z)
• Tiny Autoregressive Recursive Models [48.902738432071935]
  Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI.<n>We propose the Autoregressive TRM and evaluate it on small autoregressive tasks.<n>Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture.
  arXiv  Detail & Related papers  (2026-03-09T08:22:45Z)
• Efficient Reasoning via Reward Model [24.105621725286497]
  Reinforcement learning with verifiable rewards (RLVR) has been shown to enhance the reasoning capabilities of large language models (LLMs)<n>LRMs such as DeepSeek-R1 and OpenAI o1 often generate verbose responses containing redundant or irrelevant reasoning step-a phenomenon known as overthinking.<n>We introduce a novel reward formulation named Conciseness Reward Function (CRF) with explicit dependency between the outcome reward and conciseness score.
  arXiv  Detail & Related papers  (2025-11-12T09:51:07Z)
• CALM Before the STORM: Unlocking Native Reasoning for Optimization Modeling [60.55856973678002]
  Large Reasoning Models (LRMs) have demonstrated strong capabilities in complex multi-step reasoning.<n>Existing domain adaptation methods, originally designed for earlier instruction-tuned models, often fail to exploit the advanced reasoning patterns of modern LRMs.<n>We propose textbfCALM, a framework that progressively refines LRMs within their native reasoning modes for optimization modeling tasks.
  arXiv  Detail & Related papers  (2025-10-05T13:38:31Z)
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding [84.42056293290015]
  We analyze the token-level misalignment between reasoning and non-reasoning models.<n>Motivated by the Local Misalignment Diminish, we propose FoReaL-Decoding.<n>On four popular math-reasoning benchmarks, FoReaL-Decoding reduces theoretical FLOPs by 30 to 50% and trims CoT length by up to 40%.
  arXiv  Detail & Related papers  (2025-06-08T05:08:32Z)
• Discriminative Policy Optimization for Token-Level Reward Models [55.98642069903191]
  Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs)<n>Q-RM explicitly learns token-level Q-functions from preference data without relying on fine-grained annotations.<n>Reinforcement learning with Q-RM significantly enhances training efficiency, achieving convergence 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH.
  arXiv  Detail & Related papers  (2025-05-29T11:40:34Z)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models [50.4652276723694]
  Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities.<n>Think-RM achieves state-of-the-art results on RM-Bench, outperforming both BT RM and vertically scaled GenRM by 8%.
  arXiv  Detail & Related papers  (2025-05-22T05:56:11Z)
• RM-R1: Reward Modeling as Reasoning [81.50471199906738]
  Reasoning Reward Models (ReasRMs) formulate reward modeling as a reasoning task.<n>We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1.<n>Our models achieve state-of-the-art performance across three reward model benchmarks on average.
  arXiv  Detail & Related papers  (2025-05-05T06:11:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.