Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs

• URL: http://arxiv.org/abs/2603.27494v1
• Date: Sun, 29 Mar 2026 03:18:57 GMT
• Title: Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs
• Abstract summary: MLLMs autonomously utilize image cropping tool to analyze regions of interest for question answering.<n>We demonstrate the model's strong reliance on global input and its weak dependence on the details within the cropped region.<n>We propose a novel two-stage reinforcement learning framework that does not require supervision.
• Score: 58.32326205291745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To enhance the perception and reasoning capabilities of multimodal large language models in complex visual scenes, recent research has introduced agent-based workflows. In these works, MLLMs autonomously utilize image cropping tool to analyze regions of interest for question answering. While existing training strategies, such as those employing supervised fine-tuning and reinforcement learning, have made significant progress, our empirical analysis reveals a key limitation. We demonstrate the model's strong reliance on global input and its weak dependence on the details within the cropped region. To address this issue, we propose a novel two-stage reinforcement learning framework that does not require trajectory supervision. In the first stage, we introduce the ``Information Gap" mechanism by adjusting the granularity of the global image. This mechanism trains the model to answer questions by focusing on cropped key regions, driven by the information gain these regions provide. The second stage further enhances cropping precision by incorporating a grounding loss, using a small number of bounding box annotations. Experiments show that our method significantly enhances the model's attention to cropped regions, enabling it to achieve state-of-the-art performance on high-resolution visual question-answering benchmarks. Our method provides a more efficient approach for perceiving and reasoning fine-grained details in MLLMs. Code is available at: https://github.com/XuanPu-Z/LFPC.

Related papers

• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception [43.08943307183693]
  Region-to-Image Distillation transforms zooming from an inference-time tool into a training-time primitive.<n>We show that our models achieve leading performance across multiple fine-grained perception benchmarks.
  arXiv  Detail & Related papers  (2026-02-12T12:00:35Z)
• Image Tiling for High-Resolution Reasoning: Balancing Local Detail with Global Context [14.558348909684085]
  We present a detailed reproduction and critical analysis of the Monkey Vision-Language Model (VLM) published in CVPR24.<n>We confirm the key finding of the original Monkey VLM work, namely that tiling effectively recovers local details.<n>We then extend this work further, by investigating the effect of the inclusion of the global context.
  arXiv  Detail & Related papers  (2025-12-11T23:17:38Z)
• FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning [62.11389260206383]
  textscFineRS is a two-stage MLLM-based reinforcement learning framework for segmenting extremely small objects.<n>We present textscFineRS-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets.
  arXiv  Detail & Related papers  (2025-10-24T10:14:17Z)
• Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models [50.98559225639266]
  Sub-images with higher semantic relevance to the entire image encapsulate richer visual information for preserving the model's visual understanding ability.<n>Global Semantic-guided Weight Allocator (GSWA) module allocates weights to sub-images based on their relative information density.<n>SleighVL, a lightweight yet high-performing model, outperforms models with comparable parameters and remains competitive with larger models.
  arXiv  Detail & Related papers  (2025-01-24T06:42:06Z)
• CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models [16.91226496250909]
  We break down multi-modal understanding into two stages, from Coarse to Fine.<n>In the first stage, we prompt the MLLM to locate the approximate area of the answer.<n>In the second stage, we further enhance the model's focus on relevant areas through visual prompt engineering.
  arXiv  Detail & Related papers  (2024-12-22T05:42:40Z)
• USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation [24.90512145836643]
  We introduce a Unified Skeleton-based Dense Representation Learning framework based on feature decorrelation.<n>We show that our approach significantly outperforms the current state-of-the-art (SOTA) approaches.
  arXiv  Detail & Related papers  (2024-12-12T12:20:27Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Recognize Any Regions [55.76437190434433]
  RegionSpot integrates position-aware localization knowledge from a localization foundation model with semantic information from a ViL model.<n>Experiments in open-world object recognition show that our RegionSpot achieves significant performance gain over prior alternatives.
  arXiv  Detail & Related papers  (2023-11-02T16:31:49Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.