Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers

• URL: http://arxiv.org/abs/2606.32020v1
• Date: Tue, 30 Jun 2026 17:51:26 GMT
• Title: Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers
• Abstract summary: Bridge is a lightweight latent interface that maps Student latents into the Teacher space without modifying the Student backbone.<n>It improves SD 1.5 from 5.4 to 9.4 HPSv3 while preserving one-step inference, low latency, and broad ecosystem compatibility.
• Score: 47.916191066149224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern one-step diffusion models achieve impressive quality through distribution-based timestep distillation. Yet, they rely on a critical assumption: Teacher and Student must inhabit the same latent space. This Shared-Space constraint prevents knowledge transfer from modern high-capacity Teachers (e.g., SD 3.5 and Flux) into compact, deployment-friendly Students such as SD 1.5, whose latent resolution and VAE parameterization differ from the Teacher. We formalize this overlooked regime as Cross-Space Distillation, where Teacher and Student differ in both latent resolution and VAE space. To enable distillation under this mismatch, we introduce the Bridge, a lightweight latent interface that maps Student latents into the Teacher space without modifying the Student backbone. Bridge combines a frozen Student VAE decoder as a spatial prior with a compact learnable projector, and is trained with latent reconstruction and attention fidelity objectives for stable Teacher-space alignment. Across diverse modern Teachers, Bridge enables substantial gains for compact one-step Students; for example, it improves SD 1.5 from 5.4 to 9.4 HPSv3 while preserving one-step inference, low latency, and broad ecosystem compatibility. These results show that heterogeneous large Teachers can be distilled into efficient, deployable backbones through a lightweight latent-space interface.

Related papers

• Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models [25.503555874738623]
  Poly-OPD is a framework that can consolidate complementary strengths of heterogeneous teachers into a single compact flow-matching student.<n>Each student-generated image is re-encoded by a selected teacher's encoder and refined from a noise level matched by magnitude under the teacher's noise schedule.
  arXiv  Detail & Related papers  (2026-08-05T01:48:55Z)
• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation [26.38620865128242]
  DuoMem is a dual-space distillation framework that transfers procedural problem-solving ability from a large teacher model to compact student models.<n>It is evaluated on ALFWorld, a challenging embodied decision-making benchmark.<n>The DuoMem-enhanced 4B model completes tasks over 3x faster than the 72B teacher in wall-clock time.
  arXiv  Detail & Related papers  (2026-06-29T08:38:40Z)
• SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation [1.1617094867403568]
  Large-scale vision foundation models have driven substantial gains on dense prediction tasks such as semantic segmentation.<n>Modern foundation teachers are predominantly transformer-based that encode global context, whereas efficient students are typically convolutional networks with locally biased receptive fields.<n>Existing distillation methods largely assume architectural homogeneity and rely on direct feature mimicry, which fails to bridge this representational gap.<n>We propose SWARD, a knowledge distillation framework that addresses this gap through two complementary mechanisms.
  arXiv  Detail & Related papers  (2026-05-31T04:29:31Z)
• Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models [63.169731309799324]
  Class-incremental learning aims to continuously acquire new knowledge while preserving previously learned information, thereby mitigating catastrophic forgetting.<n>We propose HDSD, a Hierarchical Dual-Subspace Decoupling framework for continual learning in vision-language models.<n> Specifically, we introduce a lightweight Feature Modulation Module (FMM) that explicitly decomposes the parameter space into general and task-specific subspaces.
  arXiv  Detail & Related papers  (2026-05-08T09:42:05Z)
• BridgeTA: Bridging the Representation Gap in Knowledge Distillation via Teacher Assistant for Bird's Eye View Map Segmentation [17.072492774587456]
  Camera-only approaches have drawn attention as cost-effective alternatives to LiDAR, but they still fall behind LiDAR-Camera (LC) fusion-based methods.<n>We introduce BridgeTA, a cost-effective distillation framework to bridge the representation gap between LC fusion and Camera-only models.<n>Our method achieves an improvement of 4.2% mIoU over the Camera-only baseline, up to 45% higher than the improvement of other state-of-the-art KD methods.
  arXiv  Detail & Related papers  (2025-08-13T08:28:21Z)
• CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework [1.2172320168050466]
  We introduce Consensus-oriented Masked Distillation (CoMAD)<n>It unifies knowledge from self-supervised Vision Transformers into a compact student network.<n>On ImageNet-1K, CoMAD's ViT-Tiny achieves 75.4 percent Top-1, an increment of 0.4 percent over the previous state-of-the-art.
  arXiv  Detail & Related papers  (2025-08-06T18:55:14Z)
• Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation [52.0297393822012]
  We introduce an assistant model as a bridge to facilitate smooth feature knowledge transfer between heterogeneous teachers and students.<n>Within our proposed design principle, the assistant model combines the advantages of cross-architecture inductive biases and module functions.<n>Our proposed method is evaluated across some homogeneous model pairs and arbitrary heterogeneous combinations of CNNs, ViTs, spatial KDs.
  arXiv  Detail & Related papers  (2024-10-16T08:02:49Z)
• Switching Temporary Teachers for Semi-Supervised Semantic Segmentation [45.20519672287495]
  The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student.
  arXiv  Detail & Related papers  (2023-10-28T08:49:16Z)
• Cross Architecture Distillation for Face Recognition [49.55061794917994]
  We develop an Adaptable Prompting Teacher network (APT) that integrates prompts into the teacher, enabling it to manage distillation-specific knowledge. Experiments on popular face benchmarks and two large-scale verification sets demonstrate the superiority of our method.
  arXiv  Detail & Related papers  (2023-06-26T12:54:28Z)
• Faculty Distillation with Optimal Transport [53.69235109551099]
  We propose to link teacher's task and student's task by optimal transport. Based on the semantic relationship between their label spaces, we can bridge the support gap between output distributions. Experiments under various settings demonstrate the succinctness and versatility of our method.
  arXiv  Detail & Related papers  (2022-04-25T09:34:37Z)
• Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation [67.75526580926149]
  Knowledge distillation aims at obtaining a compact and effective model by learning the mapping function from a much larger one. We investigate the capacity gap problem by study the gap of confidence between teacher and student. We find that the magnitude of confidence is not necessary for knowledge distillation and could harm the student performance if the student are forced to learn confidence.
  arXiv  Detail & Related papers  (2020-10-15T03:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.