OPD-V: Visual On-Policy Self-Distillation with Modality Balance

• URL: http://arxiv.org/abs/2608.05131v2
• Date: Thu, 06 Aug 2026 08:22:41 GMT
• Title: OPD-V: Visual On-Policy Self-Distillation with Modality Balance
• Abstract summary: On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in large language models (MLLMs)<n>Existing methods draw privileged information from diverse input sources to guide self-distillation.<n>We introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher.
• Score: 60.972871141106886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM reasoning. When textual information dominates generation, the model cannot fully integrate its multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their reasoning correctness and token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy tokens used for self-distillation. Experiments across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves reasoning performance while reducing training cost.

Related papers

• Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation [7.759584418408814]
  On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters.<n>We propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian.<n>USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales.
  arXiv  Detail & Related papers  (2026-08-08T15:13:07Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning [25.860185076316508]
  Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks.<n>They may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks.<n>We propose Visual-Noise Guided In-Context Distillation (VGID) as a distillation-based framework for MLLM unlearning.
  arXiv  Detail & Related papers  (2026-05-26T12:49:34Z)
• EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation [5.310892696470208]
  On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm.<n>The challenge in this approach is that the privileged information can change model behavior more than intended.<n>We propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPD with two distinct characteristics.
  arXiv  Detail & Related papers  (2026-05-22T10:55:15Z)
• Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation [71.92541392470103]
  Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding.<n>We propose Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework.<n>Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student.
  arXiv  Detail & Related papers  (2026-05-18T17:57:04Z)
• Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training [82.17582358979884]
  Post-training with explicit reasoning traces is common to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs)<n>We propose VISTA, a vision-aware self-improvement training framework for enhancing the multimodal reasoning of MLLMs.
  arXiv  Detail & Related papers  (2026-05-12T10:44:35Z)
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models [44.041109669153506]
  On-Policy Self-Distillation (OPSD) is a framework where a single model acts as both teacher and student by conditioning on different contexts.<n>We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-01-26T17:56:50Z)
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation [67.98620973023709]
  VOLD is a framework to transfer reasoning capabilities from text-only teacher models to VLM student models.<n>We show that VOLD outperforms the baseline model significantly and improves over the state of the art by a margin.
  arXiv  Detail & Related papers  (2025-10-27T16:32:12Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.