Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

• URL: http://arxiv.org/abs/2608.21762v2
• Date: Sat, 29 Aug 2026 06:58:16 GMT
• Title: Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) fail many detail-centric questions for a concrete reason.<n>Allocating more visual tokens to every query improves some OCR and document cases, but it spends indiscriminately and can disturb tasks that rely on global context.<n>We propose GapSight, a framework for learning visual re-reading.
• Score: 9.530053392873178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image is compressed into a low-resolution global view. Allocating more visual tokens to every query improves some OCR and document cases, but it spends computation indiscriminately and can disturb tasks that rely on global context. We propose GapSight, a framework for learning visual re-reading: a VLM first takes a global glance, then selectively returns to a free-form region when the question calls for local evidence. The supervision comes from the target model's own failure signal. Offline, we compare answer loss or multiple-choice option margin under a global-only view and candidate crop-augmented views; crops that improve the target answer become model-specific review labels. A lightweight free-form crop router distills these labels into a one-shot inference policy that predicts whether to review, expected utility, and a continuous crop box from the global state. Across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct, GapSight improves the Base no-zoom baseline on six benchmarks spanning OCR, documents, charts, infographics, VStarBench, and MME-RealWorld-Lite. On InternVL2.5-8B, GapSight raises the six-benchmark average from 52.25 to 64.29, above CropVLM (57.16), ViCrop (55.84), and ZoomRefine (54.43). Mechanism analyses show that the router rescues concrete wrong answers, adapts its action rate by task, and forms a favorable token-performance profile. These results position loss-gap supervision as a practical route to teaching VLMs when and where to look again.

Related papers

• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making [30.869819258358405]
  Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images.<n>We introduce COMPLEXITYWORLD, a benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories.
  arXiv  Detail & Related papers  (2026-08-05T10:05:41Z)
• Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization [59.164665622252016]
  We introduce Cross-Modal Visual Feedback (CMVF)<n> CMVF incorporates (1) a failure visual diagnosis stage, in which a stronger VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns.<n>We show CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T12:31:58Z)
• Evaluating Vision-Language Models as a Zero-Shot Learning Alternative to You Only Look Once and Optical Character Recognition for Nigerian License Plate Recognition [0.0]
  This study explores the potential of Vision-Language Models (VLMs) as a unified, zeroshot learning solution for Nigerian license plate recognition.<n>We evaluate five selected VLMs: Gemini 2.0 Flash Exp (Google DeepMind), Qwen2.5-VL-7B-Instruct (Alibaba), GPT-4o (OpenAI), Claude 4 Sonnet (Anthropic) and Llama 3.2 Vision 90b (Meta)<n>Results based on Character Error Rate (CER) reveal that Gemini and Qwen significantly outperform other models in both accuracy and robustness, on the challenging image scenarios.
  arXiv  Detail & Related papers  (2026-07-02T10:55:24Z)
• Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap [6.097306160389619]
  ReVisIT is a train-free framework that treats each retrieved image-label pair as a unit of visual thought.<n>On VL-ICL Bench Fast Open MiniImageNet, Qwen3-VL-30B-A3B with ReVisIT reaches 98.5% at 4-shot, statistically indistinguishable from the 72B LLaVA-VisionOne SOTA (98.7%) on this near-saturated task at about 1/2.4 the parameters.
  arXiv  Detail & Related papers  (2026-07-01T08:31:09Z)
• Visual-ERM: Reward Modeling for Visual Equivalence [59.317480168347664]
  Visual Equivalence Reward Model (Visual-ERM) is a multimodal generative reward model that provides fine-grained, interpretable, and task-agnostic feedback.<n>Visual-ERM improves Qwen3-VL-8B-Instruct by +8.4 on chart-to-code and yields consistent gains on table and SVG parsing.<n>VisualCritic-RewardBench (VC-RewardBench) is a benchmark for judging fine-grained image-to-image discrepancies on structured visual data.
  arXiv  Detail & Related papers  (2026-03-13T17:58:14Z)
• Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing [76.2602505940467]
  Existing models often struggle with visually dense charts, leading to errors like data omission, misalignment, and hallucination.<n>Inspired by the human strategy of using a finger as a visual anchor'' to ensure accuracy when reading complex charts, we propose a new paradigm named Visual Self-Refine (VSR)<n>The core idea of VSR is to enable a model to generate pixel-level localization outputs, visualize them, and then feed these visualizations back to itself, allowing it to intuitively inspect and correct its own potential visual perception errors.
  arXiv  Detail & Related papers  (2026-02-18T13:40:53Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• GLEAM: Learning to Match and Explain in Cross-View Geo-Localization [66.11208984986813]
  Cross-View Geo-Localization (CVGL) focuses on identifying correspondences between images captured from distinct perspectives of the same geographical location.<n>We present GLEAM-C, a foundational CVGL model that unifies multiple views and modalities-including UAV imagery, street maps, panoramic views, and ground photographs-by aligning them exclusively with satellite imagery.<n>To address the lack of interpretability in traditional CVGL methods, we propose GLEAM-X, which combines cross-view correspondence prediction with explainable reasoning.
  arXiv  Detail & Related papers  (2025-09-09T07:14:31Z)
• RT-VLM: Re-Thinking Vision Language Model with 4-Clues for Real-World Object Recognition Robustness [2.9979091009694088]
  Real world deployments often expose modern object recognition models to domain shifts that precipitate a severe drop in accuracy.<n>To mitigate this degradation, we introduce the Re-Thinking Vision Language Model (RT-VLM) framework.
  arXiv  Detail & Related papers  (2025-09-01T02:13:00Z)
• Benchmark Granularity and Model Robustness for Image-Text Retrieval [44.045767657945895]
  We show how dataset granularity and query perturbations affect retrieval performance and robustness.<n>We show that richer captions consistently enhance retrieval, especially in text-to-image tasks.<n>Our results highlight variation in model robustness and a dataset-dependent relationship between caption granularity and sensitivity perturbation.
  arXiv  Detail & Related papers  (2024-07-21T18:08:44Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.