Visual-Advantage On-Policy Distillation for Vision-Language Models

• URL: http://arxiv.org/abs/2605.21924v1
• Date: Thu, 21 May 2026 02:48:36 GMT
• Title: Visual-Advantage On-Policy Distillation for Vision-Language Models
• Abstract summary: Visual advantage (VA) is the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail.<n>VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes.
• Score: 32.05491249378337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.

Related papers

• OPD-V: Visual On-Policy Self-Distillation with Modality Balance [60.972871141106886]
  On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in large language models (MLLMs)<n>Existing methods draw privileged information from diverse input sources to guide self-distillation.<n>We introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher.
  arXiv  Detail & Related papers  (2026-08-05T17:53:06Z)
• Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models [23.382646724389545]
  On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions.<n>We propose emphFisher-Projected On-Policy Distillation (FP-OPD), which distills only locally realizable teacher corrections.
  arXiv  Detail & Related papers  (2026-08-02T14:16:14Z)
• Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy [27.323597665236644]
  Large Vision-Language Models (LVLMs) show strong visualization interpretation.<n>It is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training.<n>We present a framework that isolates visual correctness from factual correctness.
  arXiv  Detail & Related papers  (2026-06-02T04:36:32Z)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals [65.71225905458182]
  Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
  arXiv  Detail & Related papers  (2026-05-20T03:06:36Z)
• Selective Training for Large Vision Language Models via Visual Information Gain [7.834991119179473]
  We introduce Visual Information Gain (VIG), a perplexity-based metric.<n>VIG measures the reduction in prediction uncertainty provided by visual input.<n>We propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens.
  arXiv  Detail & Related papers  (2026-02-19T09:12:21Z)
• Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy [75.66913260900726]
  Reinforcement Learning with Verifiable Rewards has significantly advanced reasoning capabilities in Large Language Models.<n>Existing paradigms, driven by text-centric outcome rewards, encourage models to bypass visual perception.<n>We propose textbfThinking with Deltas, a framework driven by a textbfDifferential Visual Reasoning Policy.
  arXiv  Detail & Related papers  (2026-01-11T08:25:34Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens [37.11253070112327]
  Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities.<n>Some priors introduce Knowledge Distillation to enhance student models, but they overlook the fundamental differences in fine-grained vision comprehension.<n>We propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation.
  arXiv  Detail & Related papers  (2025-11-26T06:45:59Z)
• Probing Visual Language Priors in VLMs [51.016683265437536]
  We introduce ViLP, a benchmark featuring deliberately out-of-distribution images.<n>Each question in ViLP is coupled with three potential answers and three corresponding images.<n>We propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training.
  arXiv  Detail & Related papers  (2024-12-31T17:54:29Z)
• Silkie: Preference Distillation for Large Visual Language Models [56.10697821410489]
  This paper explores preference distillation for large vision language models (LVLMs) We first build a vision-language feedback dataset utilizing AI annotation. We adopt GPT-4V to assess the generated outputs regarding helpfulness, visual faithfulness, and ethical considerations. The resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME benchmark regarding the perception and cognition capabilities.
  arXiv  Detail & Related papers  (2023-12-17T09:44:27Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)
• Compressing Visual-linguistic Model via Knowledge Distillation [43.73998154661652]
  We study knowledge distillation to compress a transformer-based large visual-linguistic model into a small model. We show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks.
  arXiv  Detail & Related papers  (2021-04-05T18:02:17Z)
• PatchVAE: Learning Local Latent Codes for Recognition [38.82903227239025]
  unsupervised representations learned by VAEs are significantly outperformed by those learned by supervised learning for recognition. Our hypothesis is that to learn useful representations for recognition the model needs to be encouraged to learn about repeating and consistent patterns in data. Our experiments demonstrate that representations learned by our method perform much better on the recognition tasks compared to those learned by vanilla VAEs.
  arXiv  Detail & Related papers  (2020-04-07T18:01:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.