Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2605.18740v3
• Date: Wed, 27 May 2026 17:29:28 GMT
• Title: Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
• Abstract summary: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding.<n>We propose Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework.<n>Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student.
• Score: 71.92541392470103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. We observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on the corresponding full images, suggesting that many failures stem from difficulty to focus on relevant evidence rather than insufficient local recognition ability. Motivated by this observation, we propose Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework that transfers the model's own privileged regional perception to its full-image policy. Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student. The student generates on-policy rollouts, and Vision-OPD minimizes token-level divergence between the teacher and student next-token distributions along these rollouts. This enables the model to internalize the benefit of visual zooming without external teacher models, ground-truth labels, reward verifiers, or inference-time tool use. Experiments on multiple fine-grained visual understanding benchmarks show that Vision-OPD models achieve competitive or superior performance against much larger open-source, closed-source, and "Thinking-with-Images" agentic models.

Related papers

• Towards High-Resolution Visual Perception via Hierarchical Entity Exploration [36.649153640650496]
  Hierarchical Entity Exploration transforms static image understanding into query-guided entity exploration.<n>HEE is a training-free and model-agnostic framework that transforms static image understanding into dynamic, query-guided entity exploration.
  arXiv  Detail & Related papers  (2026-07-01T11:41:21Z)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis [63.49939245944781]
  We introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images.<n>We find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge.
  arXiv  Detail & Related papers  (2026-06-17T13:23:26Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts [23.97104853350071]
  We empirically investigate emphvisual fairness in several mainstream vision-language models (LVLMs)<n>Our fairness evaluation framework employs direct and single-choice question prompt on visual question-answering/classification tasks.<n>We propose a potential multi-modal Chain-of-thought (CoT) based strategy for unfairness mitigation, applicable to both open-source and closed-source LVLMs.
  arXiv  Detail & Related papers  (2024-06-25T23:11:39Z)
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs [50.77984109941538]
  Our research reveals that the visual capabilities in recent multimodal LLMs still exhibit systematic shortcomings. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. We evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs.
  arXiv  Detail & Related papers  (2024-01-11T18:58:36Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models [50.653838482083614]
  This paper introduces a scalable test-bed to assess the capabilities of IT-LVLMs on fundamental computer vision tasks.<n> MERLIM contains over 300K image-question pairs and has a strong focus on detecting cross-modal "hallucination" events in IT-LVLMs.
  arXiv  Detail & Related papers  (2023-12-03T16:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.