Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning

• URL: http://arxiv.org/abs/2510.01681v1
• Date: Thu, 02 Oct 2025 05:14:52 GMT
• Title: Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
• Authors: Xuchen Li, Xuzhao Li, Jiahui Gao, Renjie Pi, Shiyu Hu, Wentao Zhang,
• Abstract summary: Vision-Language Models (VLMs) excel at many multimodal tasks, yet they frequently struggle with tasks requiring precise understanding and handling of fine-grained visual elements.<n>Recent work has shown promise by incorporating pixel-level visual information into the reasoning process.<n>We propose the first framework for adaptive pixel reasoning that dynamically determines necessary pixel-level operations based on the input query.
• Score: 35.475941880366726
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) excel at many multimodal tasks, yet they frequently struggle with tasks requiring precise understanding and handling of fine-grained visual elements. This is mainly due to information loss during image encoding or insufficient attention to critical regions. Recent work has shown promise by incorporating pixel-level visual information into the reasoning process, enabling VLMs to access high-resolution visual details during their thought process. However, this pixel-level information is often overused, leading to inefficiency and distraction from irrelevant visual details. To address these challenges, we propose the first framework for adaptive pixel reasoning that dynamically determines necessary pixel-level operations based on the input query. Specifically, we first apply operation-aware supervised fine-tuning to establish baseline competence in textual reasoning and visual operations, then design a novel rollout-guided reinforcement learning framework relying on feedback of the model's own responses, which enables the VLM to determine when pixel operations should be invoked based on query difficulty. Experiments on extensive multimodal reasoning benchmarks show that our model achieves superior performance while significantly reducing unnecessary visual operations. Impressively, our model achieves 73.4\% accuracy on HR-Bench 4K while maintaining a tool usage ratio of only 20.1\%, improving accuracy and simultaneously reducing tool usage by 66.5\% compared to the previous methods.

Related papers

• ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models [11.263321053154364]
  ERGO is a reasoning-driven perception-leveraging multimodal context to determine where to focus.<n>We develop simple yet effective reward components in a reinforcement learning framework for coarse-to-fine perception.<n>Our approach delivers higher accuracy than the original model and competitive methods, with greater efficiency.
  arXiv  Detail & Related papers  (2025-09-26T07:15:19Z)
• Reinforced Visual Perception with Tools [66.79840157663237]
  We introduce a novel RL algorithm based on GRPO, designed to train models to reason with a suite of four visual tools.<n>We show that our method achieves state-of-the-art performance on several perception-heavy benchmarks.<n>Our ReVPT-3B and ReVPT-7B outperform the instruct models by 9.03% and 9.44% on CV-Bench.
  arXiv  Detail & Related papers  (2025-09-01T17:57:49Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning [39.66636859076594]
  Chain-of-thought reasoning has significantly improved the performance of Large Language Models.<n>We introduce the concept of reasoning in the pixel-space.<n>We demonstrate that this approach significantly improves Vision-Language Models.
  arXiv  Detail & Related papers  (2025-05-21T19:35:08Z)
• When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning [31.696397337675847]
  Large Vision-Language Models (LVLMs) typically employ limited pre-defined grids to process images.<n>We propose a text-guided token pruning method with Dynamic Image Pyramid (DIP) integration.<n>Our method outperforms existing high-resolution strategies on four datasets using the same data.
  arXiv  Detail & Related papers  (2025-03-10T17:51:16Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding [94.64781599202882]
  Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks.<n>They often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison.<n>We propose CogAlign, a novel post-training strategy inspired by Piaget's theory of cognitive development.
  arXiv  Detail & Related papers  (2025-02-17T06:54:49Z)
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• Toward Super-Resolution for Appearance-Based Gaze Estimation [4.594159253008448]
  Super-resolution has been shown to improve image quality from a visual perspective. We propose a two-step framework based on SwinIR super-resolution model. Self-supervised learning aims to learn from unlabelled data to reduce the amount of required labeled data for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-17T17:40:32Z)
• Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
  Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
  arXiv  Detail & Related papers  (2023-03-10T18:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.