Can Contrastive Learning Improve Class-Imbalanced Diffusion Model?

• URL: http://arxiv.org/abs/2507.09052v1
• Date: Fri, 11 Jul 2025 21:58:03 GMT
• Title: Can Contrastive Learning Improve Class-Imbalanced Diffusion Model?
• Authors: Fang Chen, Alex Villa, Gongbo Liang, Xiaoyi Lu, Meng Tang,
• Abstract summary: Training data for class-conditional image synthesis often exhibit a long-tailed distribution with limited images for tail classes.<n>We introduce two deceptively simple but highly effective contrastive loss functions to improve diversity of tail class images.<n>We are the first to adapt such alignment to diffusion models.
• Score: 5.420928828677281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training data for class-conditional image synthesis often exhibit a long-tailed distribution with limited images for tail classes. Such an imbalance causes mode collapse and reduces the diversity of synthesized images for tail classes. For class-conditional diffusion models trained on imbalanced data, we aim to improve the diversity of tail class images without compromising the fidelity and diversity of head class images. We achieve this by introducing two deceptively simple but highly effective contrastive loss functions. Firstly, we employ an unsupervised InfoNCE loss utilizing negative samples to increase the distance/dissimilarity among synthetic images, particularly for tail classes. To further enhance the diversity of tail classes, our second loss is an MSE loss that contrasts class-conditional generation with unconditional generation at large timesteps. This second loss makes the denoising process insensitive to class conditions for the initial steps, which enriches tail classes through knowledge sharing from head classes. Conditional-unconditional alignment has been shown to enhance the performance of long-tailed GAN. We are the first to adapt such alignment to diffusion models. We successfully leveraged contrastive learning for class-imbalanced diffusion models. Our contrastive learning framework is easy to implement and outperforms standard DDPM and alternative methods for class-imbalanced diffusion models across various datasets, including CIFAR10/100-LT, PlacesLT, TinyImageNetLT, and ImageNetLT.

Related papers

• Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
  Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
  arXiv  Detail & Related papers  (2024-02-16T16:47:21Z)
• Generating images of rare concepts using pre-trained diffusion models [32.5337654536764]
  Text-to-image diffusion models can synthesize high-quality images, but they have various limitations. We show that their limitation is partly due to the long-tail nature of their training data. We show that rare concepts can be correctly generated by carefully selecting suitable generation seeds in the noise space.
  arXiv  Detail & Related papers  (2023-04-27T20:55:38Z)
• Improving GANs for Long-Tailed Data through Group Spectral Regularization [51.58250647277375]
  We propose a novel group Spectral Regularizer (gSR) that prevents the spectral explosion alleviating mode collapse. We find that gSR effectively combines with existing augmentation and regularization techniques, leading to state-of-the-art image generation performance on long-tailed data.
  arXiv  Detail & Related papers  (2022-08-21T17:51:05Z)
• Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data [11.66734752179563]
  Classification on long-tailed distributed data is a challenging problem. Learning on tail classes is especially challenging for the fine-tuning when transferring a pretrained model to a downstream task. We propose a two-stage fine-tuning: we first fine-tune the final layer of the pretrained model with class-balanced reweighting loss, and then we perform the standard fine-tuning.
  arXiv  Detail & Related papers  (2022-07-22T03:39:51Z)
• Improving Tail-Class Representation with Centroid Contrastive Learning [145.73991900239017]
  We propose interpolative centroid contrastive learning (ICCL) to improve long-tailed representation learning. ICCL interpolates two images from a class-agnostic sampler and a class-aware sampler, and trains the model such that the representation of the ICCL can be used to retrieve the centroids for both source classes. Our result shows a significant accuracy gain of 2.8% on the iNaturalist 2018 dataset with a real-world long-tailed distribution.
  arXiv  Detail & Related papers  (2021-10-19T15:24:48Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification [31.647639786095993]
  We propose a novel hybrid network structure composed of a supervised contrastive loss to learn image representations and a cross-entropy loss to learn classifiers. Experiments on three long-tailed classification datasets demonstrate the advantage of the proposed contrastive learning based hybrid networks in long-tailed classification.
  arXiv  Detail & Related papers  (2021-03-26T05:22:36Z)
• An Empirical Study of the Collapsing Problem in Semi-Supervised 2D Human Pose Estimation [80.02124918255059]
  Semi-supervised learning aims to boost the accuracy of a model by exploring unlabeled images. We learn two networks to mutually teach each other. The more reliable predictions on easy images in each network are used to teach the other network to learn about the corresponding hard images.
  arXiv  Detail & Related papers  (2020-11-25T03:29:52Z)
• Background Splitting: Finding Rare Classes in a Sea of Background [55.03789745276442]
  We focus on the real-world problem of training accurate deep models for image classification of a small number of rare categories. In these scenarios, almost all images belong to the background category in the dataset (>95% of the dataset is background) We demonstrate that both standard fine-tuning approaches and state-of-the-art approaches for training on imbalanced datasets do not produce accurate deep models in the presence of this extreme imbalance.
  arXiv  Detail & Related papers  (2020-08-28T23:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.