Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets

• URL: http://arxiv.org/abs/2606.29837v1
• Date: Mon, 29 Jun 2026 06:24:53 GMT
• Title: Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets
• Abstract summary: We propose a trajectory-based DD framework that jointly suppresses noise and preserves transferable knowledge without relabeling or clean subsets.<n>It comprises two complementary components: Selective Guidance Reweighting (SGR) and Teacher-Inspired Auxiliary Targets (TIAT)
• Score: 51.490743278360064
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dataset distillation (DD) condenses large corpora into compact, information-rich subsets for efficient training and reuse. However, under noisy supervision, DD risks condensing corrupted associations together with useful signals, degrading robustness. Conventional noisy-label remedies (sample selection, loss weighting, label correction) tightly couple noise estimation with model optimization, often require clean anchors, and can amplify confirmation bias-assumptions that are misaligned with DD's goal of compact, plug-and-play supervision. We therefore propose a trajectory-based DD framework that jointly suppresses noise and preserves transferable knowledge without relabeling or clean subsets. It comprises two complementary components: Selective Guidance Reweighting (SGR), which fuses global forgetting patterns (second-split forgetting) with local neighborhood consistency into a progressive reweighting scheme that prioritizes clean supervision along the teacher trajectory; and Teacher-Inspired Auxiliary Targets (TIAT), which inject auxiliary residual guidance distilled from intermediate teacher dynamics to reinforce informative signals while remaining internally consistent. Together, SGR and TIAT produce distilled datasets with cleaner and richer representations under noisy supervision. The framework is robust, label-preserving, computationally lightweight, and broadly applicable, yielding consistent gains over state-of-the-art DD baselines across symmetric, asymmetric, and real-world noise.

Related papers

• RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement [23.717085162530804]
  Text-to-image (T2I) diffusion models struggle to synthesize rare concepts involving unusual attribute-object pairings.<n>We propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process.
  arXiv  Detail & Related papers  (2026-07-06T13:49:40Z)
• A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning [2.436631469537453]
  We propose textbfSFT+RL, a two-stage robust UDA framework that integratesSupervised Fine Tuning and Reinforcement Learning.<n>textbfSFT+RL achieves average improvements of textbf10.2% in clean accuracy and textbf15.8% in adversarial robustness across three datasets.
  arXiv  Detail & Related papers  (2026-07-03T20:56:12Z)
• REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling [56.888597798431704]
  Open-World Object Detection (OWOD) requires detectors to identify previously unseen objects as unknown and incrementally incorporate them into the set of known categories.<n>Existing frameworks rely heavily on exemplar replay to mitigate catastrophic forgetting, but in some real applications, storing raw data conflicts with data access restrictions and leads to data exposure risks.<n>We propose REAL-OW, a novel rehearsal-free framework that decouples incremental knowledge through a collaborative adapter architecture.
  arXiv  Detail & Related papers  (2026-07-03T06:38:54Z)
• Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation [58.33732779555114]
  We argue that the key bottleneck is a representation-level failure caused by two coupled heterogeneities.<n>We propose Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation (SpectraMB)<n>SpectraMB achieves the best results in most evaluation settings and exhibits improved robustness under noisy interactions.
  arXiv  Detail & Related papers  (2026-06-01T15:58:13Z)
• AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment [39.63424981516754]
  Asymmetric Meta-Reflective Self-Distillation (AMR-SD)<n>We introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations.<n> Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-05-18T15:14:34Z)
• Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
  Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
  arXiv  Detail & Related papers  (2026-05-14T17:51:26Z)
• ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation [9.230247128710865]
  We propose a training-free diffusion-based framework that integrates manifold consistent guidance at every denoising timestep.<n>ManifoldGD improves representativeness, diversity, and image fidelity without requiring any model retraining.
  arXiv  Detail & Related papers  (2026-02-26T18:07:10Z)
• Improved Object-Centric Diffusion Learning with Registers and Contrastive Alignment [83.56510119503265]
  Slot Attention (SA) with pretrained diffusion models has recently shown promise for object-centric learning (OCL), but suffers from slot entanglement and weak alignment between object slots and image content.<n>We propose Contrastive Object-centric Diffusion Alignment (CODA), a simple extension that (i) employs register slots to absorb residual attention and reduce interference between object slots, and (ii) applies a contrastive alignment loss to explicitly encourage slot-image correspondence.
  arXiv  Detail & Related papers  (2026-01-03T16:10:18Z)
• Layer-wise Noise Guided Selective Wavelet Reconstruction for Robust Medical Image Segmentation [16.82406992858576]
  We proposeLayer-wise Noise-Guided Selective Wavelet Reconstruction (LNG-SWR)<n>During training, we inject small, zero-mean noise at multiple layers to learn a frequency-bias prior to steer representations away from noise-sensitive directions.<n>We then apply prior-guided selective wavelet reconstruction on the input/feature branch to achieve frequency adaptation.
  arXiv  Detail & Related papers  (2025-11-20T09:03:32Z)
• Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guarantees [11.411266656183]
  We propose a lightweight surrogate for meta-reweighting that integrates mean-centering, row shifting, and label-signed modulation.<n>Across synthetic and real noisy-label benchmarks, our method consistently outperforms strong reweighting/selection baselines.
  arXiv  Detail & Related papers  (2025-10-14T07:00:12Z)
• Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Data [67.25796812343454]
  Real-world datasets for deep learning frequently suffer from the co-occurring challenges of class imbalance and label noise.<n>We propose Dual-granularity Sinkhorn Distillation (D-SINK), a novel framework that enhances dual robustness by distilling and integrating complementary insights.<n>Experiments on benchmark datasets demonstrate that D-SINK significantly improves robustness and achieves strong empirical performance in learning from long-tailed noisy data.
  arXiv  Detail & Related papers  (2025-10-09T13:05:27Z)
• Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We propose textitRestoration Score Distillation (RSD), a principled generalization of Denoising Score Distillation (DSD)<n>RSD accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images.<n>It consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets.
  arXiv  Detail & Related papers  (2025-05-19T17:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.