AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

• URL: http://arxiv.org/abs/2605.20643v1
• Date: Wed, 20 May 2026 03:06:36 GMT
• Title: AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
• Abstract summary: Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
• Score: 65.71225905458182
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher, with the teacher being conditioned on privileged information unavailable to the student. Such information can come in different types or views, such as solutions, demonstrations, feedback, or final answers. This setup provides dense token-level feedback without relying on a separate external model, but creates a fundamental asymmetry: the teacher may rely on view-specific information that the student cannot access at inference time. Moreover, the best type of privileged information is often task-dependent, making it difficult to choose a single teacher view. In this work, we address both these challenges jointly by introducing AVSD (Adaptive-View Self-Distillation), a novel method of self-distillation with multiple privileged-information views, which reconstructs token-level supervision by separating stable cross-view consensus from view-specific residual signals. AVSD identifies the consensus signal shared across views, which provides a reliable update direction, and then selectively adds the view-specific residual signal to adjust the update magnitude when it both aligns with the consensus direction and remains proportionate to the consensus signal. Experiments on math competition benchmarks (AIME24, AIME25, and HMMT25) show that AVSD consistently outperforms both single-view self-distillation baselines and GRPO, achieving average Avg@8 gains of 3.1% and 2.2% over the strongest baselines on Qwen3-8B and Qwen3-4B, respectively. Moreover, on code-generation benchmarks (Codeforces, LiveCodeBench v6) using Qwen3-8B, AVSD outperforms the single-view self-distillation baseline by 2.4% on average.

Related papers

• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning [66.13367818892503]
  We propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation.<n> TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants.<n>It combines image and question dependence with confidence to construct a response-level reward that favors jointly grounded responses.
  arXiv  Detail & Related papers  (2026-09-09T22:54:15Z)
• Rethinking Privileged Information in On-Policy Self-Distillation [45.88028371034407]
  On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information.<n>We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model.
  arXiv  Detail & Related papers  (2026-08-18T19:42:16Z)
• Visual Contrastive Self-Distillation [61.298159957622886]
  We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
  arXiv  Detail & Related papers  (2026-07-23T17:37:37Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• Single-Channel EEG-Based Cognitive Load Assessment in Online Learning: A Hybrid Deep Learning Approach [0.0]
  We study whether a single-channel, consumer-grade EEG device can distinguish easy from difficult educational-video content.<n>We implement a hybrid CNN+LSTM+Attention model that combines the raw waveform with band-power features.
  arXiv  Detail & Related papers  (2026-07-02T07:13:01Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• Self-Distilled RLVR [57.37526213765131]
  We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
  arXiv  Detail & Related papers  (2026-04-03T15:50:07Z)
• Rethinking Self-Training Based Cross-Subject Domain Adaptation for SSVEP Classification [11.404309384526355]
  We propose a novel cross-subject domain adaptation method built upon the self-training paradigm.<n>Specifically, a Filter-Bank Euclidean Alignment (FBEA) strategy is designed to exploit frequency information from SSVEP filter banks.<n>Then, we propose a Cross-Subject Self-Training (CSST) framework consisting of two stages: Pre-Training with Adversarial Learning (PTAL), which aligns the source and target distributions, and Dual-Ensemble Self-Training (DEST), which refines pseudo-label quality.
  arXiv  Detail & Related papers  (2026-01-29T03:08:14Z)
• Towards Generalized Video Quality Assessment: A Weak-to-Strong Learning Paradigm [76.63001244080313]
  Video quality assessment (VQA) seeks to predict the perceptual quality of a video in alignment with human visual perception.<n>The dominant VQA paradigm relies on supervised training with human-labeled datasets.<n>We explore weak-to-strong (W2S) learning as a new paradigm for advancing VQA without reliance on large-scale human-labeled datasets.
  arXiv  Detail & Related papers  (2025-05-06T15:29:32Z)
• Bidirectional Semi-supervised Dual-branch CNN for Robust 3D Reconstruction of Stereo Endoscopic Images via Adaptive Cross and Parallel Supervisions [15.879059896662723]
  We propose a novel fashion of bidirectional learning between two learners, each of which can play both roles of teacher and student concurrently. Specifically, we introduce two self-supervisions, i.e., Adaptive Cross Supervision (ACS) and Adaptive Parallel Supervision (APS) The learned knowledge flows across branches along two directions: a cross direction (disparity guides distribution in ACS) and a parallel direction (disparity guides disparity in APS)
  arXiv  Detail & Related papers  (2022-10-15T13:30:41Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)
• Knowledge Distillation Meets Self-Supervision [109.6400639148393]
  Knowledge distillation involves extracting "dark knowledge" from a teacher network to guide the learning of a student network. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student.
  arXiv  Detail & Related papers  (2020-06-12T12:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.