Perception-R1: Pioneering Perception Policy with Reinforcement Learning

• URL: http://arxiv.org/abs/2504.07954v1
• Date: Thu, 10 Apr 2025 17:58:27 GMT
• Title: Perception-R1: Pioneering Perception Policy with Reinforcement Learning
• Authors: En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, Wenbing Tao,
• Abstract summary: We propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training.<n>With a standard Qwen2.5-VL-3B-Instruct, Perception-R1 achieves +4.2% on RefCOCO+, +17.9% on PixMo-Count, and notably, 31.9% AP on COCO 2017 val.
• Score: 68.13805658351944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual complexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2.5-VL-3B-Instruct, Perception-R1 achieves +4.2% on RefCOCO+, +17.9% on PixMo-Count, +4.2% on PageOCR, and notably, 31.9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.

Related papers

• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.65034908728828]
  Training large language models (LLMs) as interactive agents presents unique challenges. While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Improving RL Exploration for LLM Reasoning through Retrospective Replay [45.00643118030677]
  We propose a novel algorithm named Retrospective Replay-based Reinforcement Learning (RRL), which introduces a dynamic replay mechanism throughout the training process. RRL enables the model to revisit promising states identified in the early stages, thereby improving its efficiency and effectiveness in exploration.
  arXiv  Detail & Related papers  (2025-04-19T17:40:04Z)
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model [29.524164786422368]
  Recently, DeepSeek R1 has shown that reinforcement learning can substantially improve the reasoning capabilities of Large Language Models (LLMs)<n>We investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs)<n>We develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks.
  arXiv  Detail & Related papers  (2025-04-10T10:05:15Z)
• One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF [2.1212179660694104]
  We primarily examine a variety of RL-based and RL-free methods designed to address Reinforcement Learning from Human Feedback (RLHF) and Large Reasoning Models (LRMs) We reinterpret several RL-based and RL-free algorithms through the perspective of neural structured bandit prediction. This leads to a detailed derivation of the standard RLHF objective within a full RL context, demonstrating its equivalence to neural structured bandit prediction.
  arXiv  Detail & Related papers  (2025-03-25T10:23:26Z)
• Think or Not Think: A Study of Explicit Thinking inRule-Based Visual Reinforcement Fine-Tuning [8.665713419757061]
  Rule-based reinforcement learning (RL) fine-tuning for visual classification using multi-modal large language models (MLLMs) and the role of the thinking process is investigated.<n>We introduce textitNo-Thinking-RL, a novel approach that minimizes the model's thinking during fine-tuning by utilizing an equality accuracy reward.
  arXiv  Detail & Related papers  (2025-03-20T14:37:45Z)
• DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding [61.26026947423187]
  Human experts excel at fine-grained visual discrimination by leveraging domain knowledge to refine perceptual features.<n>Current Multimodal Large Language Models (MLLMs) struggle to integrate reasoning into visual perception.<n>We propose DeepPerception, an MLLM enhanced with cognitive visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-17T04:06:34Z)
• GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training [62.536191233049614]
  Reinforcement learning with verifiable outcome rewards (RLVR) has effectively scaled up chain-of-thought (CoT) reasoning in large language models (LLMs)<n>This work investigates this problem through extensive experiments on complex card games, such as 24 points, and embodied tasks from ALFWorld.<n>We find that when rewards are based solely on action outcomes, RL fails to incentivize CoT reasoning in VLMs, instead leading to a phenomenon we termed thought collapse.
  arXiv  Detail & Related papers  (2025-03-11T15:17:02Z)
• Knowledge Graph Reasoning with Self-supervised Reinforcement Learning [30.359557545737747]
  We propose a self-supervised pre-training method to warm up the policy network before the RL training stage. In our supervised learning stage, the agent selects actions based on the policy network and learns from generated labels. We show that our SSRL model meets or exceeds current state-of-the-art results on all Hits@k and mean reciprocal rank (MRR) metrics.
  arXiv  Detail & Related papers  (2024-05-22T13:39:33Z)
• Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning [69.19840497497503]
  It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents. We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2023-09-04T09:09:54Z)
• Light-weight probing of unsupervised representations for Reinforcement Learning [20.638410483549706]
  We study whether linear probing can be a proxy evaluation task for the quality of unsupervised RL representation. We show that the probing tasks are strongly rank correlated with the downstream RL performance on the Atari100k Benchmark. This provides a more efficient method for exploring the space of pretraining algorithms and identifying promising pretraining recipes.
  arXiv  Detail & Related papers  (2022-08-25T21:08:01Z)
• Learning to Prune Deep Neural Networks via Reinforcement Learning [64.85939668308966]
  PuRL is a deep reinforcement learning based algorithm for pruning neural networks. It achieves sparsity and accuracy comparable to current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-09T13:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.