Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking

• URL: http://arxiv.org/abs/2505.18495v1
• Date: Sat, 24 May 2025 04:16:40 GMT
• Title: Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
• Authors: Chen-Hao Chao, Wei-Fang Sun, Hanwen Liang, Chun-Yi Lee, Rahul G. Krishnan,
• Abstract summary: Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence.<n>We propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states between the masked and unmasked states.<n>Our method demonstrates superior performance across a diverse set of generative modeling tasks.
• Score: 17.371579113481644
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence. Each token can take one of two states: masked or unmasked. We observe that token sequences often remain unchanged between consecutive sampling steps; consequently, the model repeatedly processes identical inputs, leading to redundant computation. To address this inefficiency, we propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states interpolated between the masked and unmasked states. This design enables the model to make predictions based on partially observed token information, and facilitates a fine-grained denoising process. We derive a variational training objective and introduce a simple architectural design to accommodate intermediate-state inputs. Our method demonstrates superior performance across a diverse set of generative modeling tasks. On text data, it achieves a perplexity of 15.36 on OpenWebText, outperforming previous MDM (21.52), autoregressive models (17.54), and their hybrid variants (17.58), without relying on an autoregressive formulation. On image data, it attains competitive FID scores of 3.26 on CIFAR-10 and 6.98 on ImageNet-32, comparable to leading continuous generative models.

Related papers

• AliTok: Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model [59.065471969232284]
  We propose a novel Aligned Tokenizer (AliTok) to align the tokenizer and autoregressive model.<n>On ImageNet-256 benchmark, using a standard decoder-only autoregressive model as the generator, AliTok achieves a gFID score of 1.50 and an IS of 305.9.<n>When the parameter count is increased to 662M, AliTok achieves a gFID score of 1.35, surpassing the state-of-the-art diffusion method with 10x faster sampling speed.
  arXiv  Detail & Related papers  (2025-06-05T17:45:10Z)
• Partition Generative Modeling: Masked Modeling Without Masks [1.4110007887109783]
  Partition Generative Models (PGMs) is a novel approach to masked generative modeling (MGMs)<n>Experiments on OpenWebText with a context length of 1024 tokens demonstrate that PGMs deliver at least 5x improvements in both latency and throughput.
  arXiv  Detail & Related papers  (2025-05-24T21:44:32Z)
• [MASK] is All You Need [28.90875822599164]
  We propose using discrete-state models to connect Masked Generative and Non-autoregressive Diffusion models.<n>By leveraging [MASK] in discrete-state models, we can bridge Masked Generative and Non-autoregressive Diffusion models.
  arXiv  Detail & Related papers  (2024-12-09T18:59:56Z)
• ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders [53.3185750528969]
  Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework. We introduce a data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise. We demonstrate our strategy's superiority in downstream tasks compared to random masking.
  arXiv  Detail & Related papers  (2024-07-17T22:04:00Z)
• MEGA: Masked Generative Autoencoder for Human Mesh Recovery [33.26995842920877]
  Human Mesh Recovery from a single RGB image is a highly ambiguous problem.<n>Most HMR methods overlook this issue and make a single prediction without accounting for this ambiguity.<n>This work proposes a new approach based on masked generative modeling.
  arXiv  Detail & Related papers  (2024-05-29T07:40:31Z)
• DFormer: Diffusion-guided Transformer for Universal Image Segmentation [86.73405604947459]
  The proposed DFormer views universal image segmentation task as a denoising process using a diffusion model. At inference, our DFormer directly predicts the masks and corresponding categories from a set of randomly-generated masks. Our DFormer outperforms the recent diffusion-based panoptic segmentation method Pix2Seq-D with a gain of 3.6% on MS COCO val 2017 set.
  arXiv  Detail & Related papers  (2023-06-06T06:33:32Z)
• Efficient Masked Autoencoders with Self-Consistency [34.7076436760695]
  Masked image modeling (MIM) has been recognized as a strong self-supervised pre-training method in computer vision. We propose efficient masked autoencoders with self-consistency (EMAE) to improve the pre-training efficiency. EMAE consistently obtains state-of-the-art transfer ability on a variety of downstream tasks, such as image classification, object detection, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-02-28T09:21:12Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Extreme Masking for Learning Instance and Distributed Visual Representations [50.152264456036114]
  The paper presents a scalable approach for learning distributed representations over individual tokens and a holistic instance representation simultaneously. We use self-attention blocks to represent distributed tokens, followed by cross-attention blocks to aggregate the holistic instance. Our model, named ExtreMA, follows the plain BYOL approach where the instance representation from the unmasked subset is trained to predict that from the intact input.
  arXiv  Detail & Related papers  (2022-06-09T17:59:43Z)
• SODAR: Segmenting Objects by DynamicallyAggregating Neighboring Mask Representations [90.8752454643737]
  Recent state-of-the-art one-stage instance segmentation model SOLO divides the input image into a grid and directly predicts per grid cell object masks with fully-convolutional networks. We observe SOLO generates similar masks for an object at nearby grid cells, and these neighboring predictions can complement each other as some may better segment certain object part. Motivated by the observed gap, we develop a novel learning-based aggregation method that improves upon SOLO by leveraging the rich neighboring information.
  arXiv  Detail & Related papers  (2022-02-15T13:53:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.