Canonical Latent Representations in Conditional Diffusion Models

• URL: http://arxiv.org/abs/2506.09955v1
• Date: Wed, 11 Jun 2025 17:28:52 GMT
• Title: Canonical Latent Representations in Conditional Diffusion Models
• Authors: Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine Süsstrunk,
• Abstract summary: Conditional diffusion models (CDMs) have shown impressive performance across a range of generative tasks.<n>We propose a novel diffusion-based feature-distillation paradigm, CaDistill.<n>Our findings suggest that CDMs can serve not just as image generators but also as compact, interpretable teachers.
• Score: 28.976069168646635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conditional diffusion models (CDMs) have shown impressive performance across a range of generative tasks. Their ability to model the full data distribution has opened new avenues for analysis-by-synthesis in downstream discriminative learning. However, this same modeling capacity causes CDMs to entangle the class-defining features with irrelevant context, posing challenges to extracting robust and interpretable representations. To this end, we identify Canonical LAtent Representations (CLAReps), latent codes whose internal CDM features preserve essential categorical information while discarding non-discriminative signals. When decoded, CLAReps produce representative samples for each class, offering an interpretable and compact summary of the core class semantics with minimal irrelevant details. Exploiting CLAReps, we develop a novel diffusion-based feature-distillation paradigm, CaDistill. While the student has full access to the training set, the CDM as teacher transfers core class knowledge only via CLAReps, which amounts to merely 10 % of the training data in size. After training, the student achieves strong adversarial robustness and generalization ability, focusing more on the class signals instead of spurious background cues. Our findings suggest that CDMs can serve not just as image generators but also as compact, interpretable teachers that can drive robust representation learning.

Related papers

• DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning [53.27049077100897]
  generative pre-training has been shown to yield discriminative representations, paving the way towards unified visual generation and understanding.<n>This work introduces self-conditioning, a mechanism that internally leverages the rich semantics inherent in denoising network to guide its own decoding layers.<n>Results are compelling: our method boosts both generation FID and recognition accuracy with 1% computational overhead and generalizes across diverse diffusion architectures.
  arXiv  Detail & Related papers  (2025-05-16T08:47:16Z)
• Automated Learning of Semantic Embedding Representations for Diffusion Models [1.688134675717698]
  We employ a multi-level denoising autoencoder framework to expand the representation capacity of denoising diffusion models.<n>Our work justifies that DDMs are not only suitable for generative tasks, but also potentially advantageous for general-purpose deep learning applications.
  arXiv  Detail & Related papers  (2025-05-09T02:10:46Z)
• Discriminative Image Generation with Diffusion Models for Zero-Shot Learning [53.44301001173801]
  We present DIG-ZSL, a novel Discriminative Image Generation framework for Zero-Shot Learning.<n>We learn a discriminative class token (DCT) for each unseen class under the guidance of a pre-trained category discrimination model (CDM)<n>In this paper, the extensive experiments and visualizations on four datasets show that our DIG-ZSL: (1) generates diverse and high-quality images, (2) outperforms previous state-of-the-art nonhuman-annotated semantic prototype-based methods by a large margin, and (3) achieves comparable or better performance than baselines that leverage human-annot
  arXiv  Detail & Related papers  (2024-12-23T02:18:54Z)
• A Dual-Fusion Cognitive Diagnosis Framework for Open Student Learning Environments [10.066184572184627]
  This paper proposes a dual-fusion cognitive diagnosis framework (DFCD) to address the challenge of aligning two different modalities. Experiments show that DFCD achieves superior performance by integrating different modalities and strong adaptability in open student learning environments.
  arXiv  Detail & Related papers  (2024-10-19T10:12:02Z)
• MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion [14.907473847787541]
  We propose Masked Diffusion Conditional (MacDiff) as a unified framework for human skeleton modeling. For the first time, we leverage diffusion models as effective skeleton representation learners. MacDiff achieves state-of-the-art performance on representation learning benchmarks while maintaining the competence for generative tasks.
  arXiv  Detail & Related papers  (2024-09-16T17:06:10Z)
• Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification [49.41632476658246]
  We discuss the extension of DFKD to Vision-Language Foundation Models without access to the billion-level image-text datasets. The objective is to customize a student model for distribution-agnostic downstream tasks with given category concepts. We propose three novel Prompt Diversification methods to encourage image synthesis with diverse styles.
  arXiv  Detail & Related papers  (2024-07-21T13:26:30Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors [56.82596340418697]
  We propose a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.
  arXiv  Detail & Related papers  (2024-01-29T10:36:57Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Representation Learning with Diffusion Models [0.0]
  Diffusion models (DMs) have achieved state-of-the-art results for image synthesis tasks as well as density estimation. We introduce a framework for learning such representations with diffusion models (LRDM) In particular, the DM and the representation encoder are trained jointly in order to learn rich representations specific to the generative denoising process.
  arXiv  Detail & Related papers  (2022-10-20T07:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.