ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models

• URL: http://arxiv.org/abs/2509.21991v1
• Date: Fri, 26 Sep 2025 07:15:19 GMT
• Title: ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
• Authors: Jewon Lee, Wooksu Shin, Seungmin Yang, Ki-Ung Song, DongUk Lim, Jaeyeon Kim, Tae-Ho Kim, Bo-Kyeong Kim,
• Abstract summary: ERGO is a reasoning-driven perception-leveraging multimodal context to determine where to focus.<n>We develop simple yet effective reward components in a reinforcement learning framework for coarse-to-fine perception.<n>Our approach delivers higher accuracy than the original model and competitive methods, with greater efficiency.
• Score: 11.263321053154364
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Efficient processing of high-resolution images is crucial for real-world vision-language applications. However, existing Large Vision-Language Models (LVLMs) incur substantial computational overhead due to the large number of vision tokens. With the advent of "thinking with images" models, reasoning now extends beyond text to the visual domain. This capability motivates our two-stage "coarse-to-fine" reasoning pipeline: first, a downsampled image is analyzed to identify task-relevant regions; then, only these regions are cropped at full resolution and processed in a subsequent reasoning stage. This approach reduces computational cost while preserving fine-grained visual details where necessary. A major challenge lies in inferring which regions are truly relevant to a given query. Recent related methods often fail in the first stage after input-image downsampling, due to perception-driven reasoning, where clear visual information is required for effective reasoning. To address this issue, we propose ERGO (Efficient Reasoning & Guided Observation) that performs reasoning-driven perception-leveraging multimodal context to determine where to focus. Our model can account for perceptual uncertainty, expanding the cropped region to cover visually ambiguous areas for answering questions. To this end, we develop simple yet effective reward components in a reinforcement learning framework for coarse-to-fine perception. Across multiple datasets, our approach delivers higher accuracy than the original model and competitive methods, with greater efficiency. For instance, ERGO surpasses Qwen2.5-VL-7B on the V* benchmark by 4.7 points while using only 23% of the vision tokens, achieving a 3x inference speedup. The code and models can be found at: https://github.com/nota-github/ERGO.

Related papers

• Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images [53.373427633330515]
  We propose DRIM, a model that enables deep but reliable multi-turn reasoning when thinking with images in its multimodal CoT.<n>Based on a high-resolution image dataset, we construct high-difficulty and verifiable visual question-answer pairs.<n>In the SFT stage, we collect tool trajectories as cold-start data, guiding a multi-turn reasoning pattern.<n>In the RL stage, we introduce redundancy-penalized policy optimization, which incentivizes the model to develop a self-reflective reasoning pattern.
  arXiv  Detail & Related papers  (2025-12-19T07:44:43Z)
• Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark [112.46338388724116]
  We introduce the Visual Reasoning Tracer (VRT) task, which requires models to not only localize the target object but also explicitly predict the intermediate objects that form the reasoning path.<n>To advance research in this area, we contribute: (1) VRT-Bench, a human-annotated benchmark for evaluating visual reasoning; (2) a new metric for assessing the quality of reasoning traces; and (3) VRT-80k, a large-scale dataset for reasoning model training.
  arXiv  Detail & Related papers  (2025-12-04T18:55:34Z)
• Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning [35.475941880366726]
  Vision-Language Models (VLMs) excel at many multimodal tasks, yet they frequently struggle with tasks requiring precise understanding and handling of fine-grained visual elements.<n>Recent work has shown promise by incorporating pixel-level visual information into the reasoning process.<n>We propose the first framework for adaptive pixel reasoning that dynamically determines necessary pixel-level operations based on the input query.
  arXiv  Detail & Related papers  (2025-10-02T05:14:52Z)
• CoFFT: Chain of Foresight-Focus Thought for Visual Language Models [61.34272727005052]
  Chain of Foresight-Focus Thought (CoFFT) is a training-free approach that enhances visual reasoning by emulating human visual cognition.<n>These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning.<n> Empirical results across multiple benchmarks using Qwen2.5-VL, InternVL-2.5, and Llava-Next demonstrate consistent performance improvements of 3.1-5.8% with controllable increasing computational overhead.
  arXiv  Detail & Related papers  (2025-09-26T07:46:30Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering [5.840924060437216]
  We propose a training-free visual cropping method, dubbed FOCUS, to guide the search for the most relevant image region.<n> FOCUS achieves strong performance across four fine-grained VQA datasets and two types of MLLMs.<n>It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye.
  arXiv  Detail & Related papers  (2025-06-26T18:51:04Z)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning [66.84770041828462]
  Recent strides in multimodal large language models (MLLMs) have significantly advanced their performance in many reasoning tasks.<n> Abstract Visual Reasoning (AVR) remains a critical challenge, primarily due to limitations in perceiving abstract graphics.<n>We propose VisuRiddles, a benchmark for PRS, featuring tasks meticulously constructed to assess models' reasoning capacities.<n>Second, we introduce the Perceptual Riddle Synthesizer (PRS), an automated framework for generating riddles with fine-grained perceptual descriptions.
  arXiv  Detail & Related papers  (2025-06-03T07:24:00Z)
• Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models [58.64449765678416]
  We introduce landscape of thoughts (LoT) to inspect the reasoning trajectories with certain reasoning methods on any multi-choice dataset.<n>LoT distinguishes between strong and weak models, correct and incorrect answers, as well as different reasoning tasks.<n>We showcase this advantage by adapting LoT to a lightweight verifier that evaluates the correctness of trajectories.
  arXiv  Detail & Related papers  (2025-03-28T06:09:51Z)
• Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning [53.790502697674754]
  We propose Take-along Visual Conditioning (TVC), a strategy that shifts image input to critical reasoning stages.<n>TVC helps the model retain attention to the visual components throughout the reasoning.<n>Our approach achieves state-of-the-art performance on average across five mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-03-17T16:45:12Z)
• A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning [9.786907179872815]
  The potential of vision and language remains underexplored in face forgery detection. There is a need for a methodology that converts face forgery detection to a Visual Question Answering (VQA) task. We propose a multi-staged approach that diverges from the traditional binary decision paradigm to address this gap.
  arXiv  Detail & Related papers  (2024-10-01T08:16:40Z)
• Addressing a fundamental limitation in deep vision models: lack of spatial attention [43.37813040320147]
  The aim of this manuscript is to underscore a significant limitation in current deep learning models, particularly vision models. Unlike human vision, which efficiently selects only the essential visual areas for further processing, deep vision models process the entire image. We propose two solutions that could pave the way for the next generation of more efficient vision models.
  arXiv  Detail & Related papers  (2024-07-01T20:21:09Z)
• Learning 1D Causal Visual Representation with De-focus Attention Networks [108.72931590504406]
  This paper explores the feasibility of representing images using 1D causal modeling. We propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.