Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings

• URL: http://arxiv.org/abs/2509.22925v1
• Date: Fri, 26 Sep 2025 20:51:20 GMT
• Title: Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• Authors: Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton,
• Abstract summary: One-step generators distilled from Masked Diffusion Models (MDMs) compress multiple sampling steps into a single forward pass.<n>They inherit modeling bias from the teacher, and their discrete token outputs block gradient flow.<n>We introduce soft embeddings, a simple relaxation that replaces discrete tokens with the expected embeddings under the generator's output distribution.
• Score: 35.979608265594685
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One-step generators distilled from Masked Diffusion Models (MDMs) compress multiple sampling steps into a single forward pass, enabling efficient text and image synthesis. However, they suffer two key limitations: they inherit modeling bias from the teacher, and their discrete token outputs block gradient flow, preventing post-distillation refinements such as adversarial training, reward-based fine-tuning, and Test-Time Embedding Optimization (TTEO). In this work, we introduce soft embeddings, a simple relaxation that replaces discrete tokens with the expected embeddings under the generator's output distribution. Soft embeddings preserve representation fidelity for one-step discrete generator while providing a fully differentiable continuous surrogate that is compatible with teacher backbones and tokenizer decoders. Integrating soft embeddings into the Di[M]O distillation framework (denoted Soft-Di[M]O) makes one-step generators end-to-end trainable and enables straightforward application of GAN-based refinement, differentiable reward fine-tuning, and TTEO. Empirically, across multiple MDM teachers (e.g., MaskBit, MaskGen), Soft-Di[M]O achieves state-of-the-art one-step results: improved class-to-image performance, a one-step FID of 1.56 on ImageNet-256 with GAN-based refinement, along with higher GenEval and HPS scores on text-to-image with reward fine-tuning, and further gains from TTEO.

Related papers

• SFTok: Bridging the Performance Gap in Discrete Tokenizers [72.9996757048065]
  We propose textbfSFTok, a discrete tokenizer that incorporates a multi-step iterative mechanism for precise reconstruction.<n>At a high compression rate of only 64 tokens per image, SFTok achieves state-of-the-art reconstruction quality on ImageNet.
  arXiv  Detail & Related papers  (2025-12-18T18:59:04Z)
• CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation [9.628074306577851]
  Current conditional autoregressive image generation methods have shown promising results, yet their potential remains largely unexplored in the practical unsupervised image translation domain.<n>A critical limitation stems from the discrete quantization inherent in traditional Vector Quantization-based frameworks.<n>We propose Softmax Relaxed Quantization, a novel approach that reformulates codebook selection as a continuous probability mixing process.
  arXiv  Detail & Related papers  (2025-06-29T17:43:04Z)
• Few-Step Diffusion via Score identity Distillation [67.07985339442703]
  Diffusion distillation has emerged as a promising strategy for accelerating text-to-image (T2I) diffusion models.<n>Existing methods rely on real or teacher-synthesized images to perform well when distilling high-resolution T2I diffusion models.<n>We propose two new guidance strategies: Zero-CFG, which disables CFG in the teacher and removes text conditioning in the fake score network, and Anti-CFG, which applies negative CFG in the fake score network.
  arXiv  Detail & Related papers  (2025-05-19T03:45:16Z)
• Di$\mathtt{[M]}$O: Distilling Masked Diffusion Models into One-step Generator [22.88494918435088]
  Masked Diffusion Models (MDMs) have emerged as a powerful generative modeling technique.<n>We propose Di$mathtt[M]$O, a novel approach that distills masked diffusion models into a one-step generator.<n>We show Di$mathtt[M]$O's effectiveness on both class-conditional and text-conditional image generation.
  arXiv  Detail & Related papers  (2025-03-19T17:36:54Z)
• Unified Auto-Encoding with Masked Diffusion [15.264296748357157]
  We propose a unified self-supervised objective, dubbed Unified Masked Diffusion (UMD) UMD combines patch-based and noise-based corruption techniques within a single auto-encoding framework. It achieves strong performance in downstream generative and representation learning tasks.
  arXiv  Detail & Related papers  (2024-06-25T16:24:34Z)
• Improved Distribution Matching Distillation for Fast Image Synthesis [54.72356560597428]
  We introduce DMD2, a set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
  arXiv  Detail & Related papers  (2024-05-23T17:59:49Z)
• Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion [35.21106030549071]
  Diffusion Probabilistic Models (DPMs) are dominant force in text-to-image generation tasks. We propose an alternative view of state-of-the-art DPMs as a way of inverting advanced Vision-Language Models (VLMs) By directly optimizing images with the supervision of discriminative VLMs, the proposed method can potentially achieve a better text-image alignment.
  arXiv  Detail & Related papers  (2024-02-26T05:08:40Z)
• Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuation [53.04220377034574]
  We propose incorporating an analytical image attenuation process into the forward diffusion process for high-quality (un)conditioned image generation.<n>Our method represents the forward image-to-noise mapping as simultaneous textitimage-to-zero mapping and textitzero-to-noise mapping.<n>We have conducted experiments on unconditioned image generation, textite.g., CIFAR-10 and CelebA-HQ-256, and image-conditioned downstream tasks such as super-resolution, saliency detection, edge detection, and image inpainting.
  arXiv  Detail & Related papers  (2023-06-23T18:08:00Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.