Improving Vision-language Models with Perception-centric Process Reward Models

• URL: http://arxiv.org/abs/2604.24583v1
• Date: Mon, 27 Apr 2026 15:08:02 GMT
• Title: Improving Vision-language Models with Perception-centric Process Reward Models
• Abstract summary: We propose Perceval, a process reward model (PRM) that enables token-level error grounding.<n>Perceval is trained with perception-intensive supervised training data.<n>We then integrate Perceval into the RL training process to train the policy models.
• Score: 94.51962346430248
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.

Related papers

• Falcon Perception-HD: High Density Perception via Reinforcement Learning [17.45540959731451]
  Post-training reinforcement learning can align autoregressive perception models with their evaluation metrics.<n>We develop two hybrid self-annotation pipelines, respectively tailored for difficult referring expressions and very dense scenes.<n>We show that a simple reward that penalizes false negatives and positives is sufficient.
  arXiv  Detail & Related papers  (2026-08-19T13:01:00Z)
• Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation [0.0]
  Reinforcement Learning is commonly used to train large language models using environmental feedback.<n>We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode.<n>When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt.
  arXiv  Detail & Related papers  (2026-07-06T18:24:16Z)
• Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning [40.28153252530189]
  Large vision-language models can reason over multimodal inputs by generating textual chains of thought.<n>A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors.<n>Existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections.<n>We propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection.
  arXiv  Detail & Related papers  (2026-07-02T17:53:15Z)
• Reinforcement Learning via Self-Distillation [37.078107691613155]
  Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math.<n>Current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck.<n>We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO)<n>SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model.
  arXiv  Detail & Related papers  (2026-01-28T17:45:12Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models [61.78513830395669]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs)<n>As models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal.<n>We propose the Explore Residual Prompts in Policy Optimization framework, which encourages exploration on residual prompts and reactivates their training signals.
  arXiv  Detail & Related papers  (2025-11-06T20:40:27Z)
• Unleashing Perception-Time Scaling to Multimodal Reasoning Models [60.578179197783754]
  Recent advances in inference-time scaling have substantially enhanced the reasoning capabilities of Large Vision-Language Models (LVLMs)<n>Inspired by this success, similar strategies have been applied to multimodal reasoning, yet their impact on visual perception remains unclear.<n>We propose Perception-Time Scaling (PTS), a novel paradigm that encourages token-rich perception and decomposes complex perception problems into intermediate tractable sub-problems.
  arXiv  Detail & Related papers  (2025-10-10T03:17:52Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs [32.13266235550995]
  Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs)<n>Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates.
  arXiv  Detail & Related papers  (2025-09-22T13:00:35Z)
• Incentivizing LLMs to Self-Verify Their Answers [22.387551134333084]
  We propose a framework that incentivizes Large Language Models to self-verify their own answers.<n>We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B.<n> Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling.
  arXiv  Detail & Related papers  (2025-06-02T06:54:29Z)
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles [91.88062410741833]
  We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning.<n>We show that OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)
• CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning [61.21923643289266]
  Chain of Manipulations is a mechanism that enables Vision-Language Models to solve problems step-by-step with evidence.<n>After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) actively without involving external tools.<n>Our trained model, textbfCogCoM, achieves state-of-the-art performance across 9 benchmarks from 4 categories.
  arXiv  Detail & Related papers  (2024-02-06T18:43:48Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.