Masked Generative Modeling with Enhanced Sampling Scheme

• URL: http://arxiv.org/abs/2309.07945v1
• Date: Thu, 14 Sep 2023 09:42:13 GMT
• Title: Masked Generative Modeling with Enhanced Sampling Scheme
• Abstract summary: Enhanced Sampling Scheme (ESS) ensures both sample diversity and fidelity. ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling. We demonstrate significant performance gains of ESS in both unconditional sampling and class-conditional sampling.
• Score: 1.3927943269211591
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a novel sampling scheme for masked non-autoregressive generative modeling. We identify the limitations of TimeVQVAE, MaskGIT, and Token-Critic in their sampling processes, and propose Enhanced Sampling Scheme (ESS) to overcome these limitations. ESS explicitly ensures both sample diversity and fidelity, and consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling. ESS starts by sampling a token set using the naive iterative decoding as proposed in MaskGIT, ensuring sample diversity. Then, the token set undergoes the critical reverse sampling, masking tokens leading to unrealistic samples. After that, critical resampling reconstructs masked tokens until the final sampling step is reached to ensure high fidelity. Critical resampling uses confidence scores obtained from a self-Token-Critic to better measure the realism of sampled tokens, while critical reverse sampling uses the structure of the quantized latent vector space to discover unrealistic sample paths. We demonstrate significant performance gains of ESS in both unconditional sampling and class-conditional sampling using all the 128 datasets in the UCR Time Series archive.

Related papers

• NI Sampling: Accelerating Discrete Diffusion Sampling by Token Order Optimization [27.77541110403583]
  We study the problem of token sampling order optimization and demonstrate its significant potential for acceleration.<n>We propose a Neural Indicator Sampling framework that utilize a neural indicator to decide which tokens should be sampled at each step.<n>Our method consistently outperforms confidence threshold sampling in the accuracy-step trade-off.
  arXiv  Detail & Related papers  (2026-04-20T16:22:59Z)
• Sample Is Feature: Beyond Item-Level, Toward Sample-Level Tokens for Unified Large Recommender Models [14.332200648147863]
  We propose textbfSIF (emphSample Is Feature), which encodes each historical Raw Sample directly into the sequence token.<n>The textbfSample Tokenizer quantizes each historical Raw Sample into a Token Sample via hierarchical group-adaptive quantization.<n>The textbfSIF-Mixer performs deep feature interaction over the homogeneous sample representations via token-level and sample-level mixing, fully unleashing the model's representational capacity.
  arXiv  Detail & Related papers  (2026-04-17T02:47:31Z)
• Wedge Sampling: Efficient Tensor Completion with Nearly-Linear Sample Complexity [9.42598427201735]
  We introduce Wedge Sampling, a new non-adaptive sampling scheme for low-rank tensor completion.<n>We study recovery of an order-$k low-rank tensor of dimension $n times cdots times n$ from a subset of its entries.
  arXiv  Detail & Related papers  (2026-02-05T16:47:13Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Combating Noisy Labels through Fostering Self- and Neighbor-Consistency [120.4394402099635]
  Label noise is pervasive in various real-world scenarios, posing challenges in supervised deep learning.<n>We propose a noise-robust method named Jo-SNC (textbfJoint sample selection and model regularization based on textbfSelf- and textbfNeighbor-textbfConsistency)<n>We design a self-adaptive, data-driven thresholding scheme to adjust per-class selection thresholds.
  arXiv  Detail & Related papers  (2026-01-19T07:55:29Z)
• One-Step Diffusion Samplers via Self-Distillation and Deterministic Flow [24.67443222055996]
  Existing sampling algorithms typically require many iterative steps to produce high-quality samples.<n>We introduce one-step diffusion samplers which learn a step-conditioned ODE.<n>We show that standard ELBO estimates degrade in the few-step regime because common discrete yield mismatched forward/backward transition kernels.
  arXiv  Detail & Related papers  (2025-12-04T20:57:53Z)
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking [17.511240770486452]
  Masked diffusion models (MDMs) have shown competitive performance compared to autoregressive models (ARMs) for language modeling.<n>We introduce EB-Sampler, a drop-in replacement for existing samplers, utilizing an Entropy Bounded unmasking procedure.<n> EB-Sampler accelerates sampling from current state of the art MDMs by roughly 2-3x on standard coding and math reasoning benchmarks without loss in performance.
  arXiv  Detail & Related papers  (2025-05-30T17:52:55Z)
• Text Generation Beyond Discrete Token Sampling [75.96920867382859]
  Mixture of Inputs (MoI) is a training-free method for autoregressive generation.<n>MoI consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B.
  arXiv  Detail & Related papers  (2025-05-20T18:41:46Z)
• Quantizing Diffusion Models from a Sampling-Aware Perspective [43.95032520555463]
  We propose a sampling-aware quantization strategy, wherein a Mixed-Order Trajectory Alignment technique is devised.<n>Experiments on sparse-step fast sampling across multiple datasets demonstrate that our approach preserves the rapid convergence characteristics of high-speed samplers.
  arXiv  Detail & Related papers  (2025-05-04T20:50:44Z)
• FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling [59.8051705468084]
  Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models. We present FR-Spec, a frequency-ranked speculative sampling framework that optimize draft candidate selection through vocabulary space compression.
  arXiv  Detail & Related papers  (2025-02-20T18:58:10Z)
• Quasi-random Multi-Sample Inference for Large Language Models [1.647759094903376]
  Large language models (LLMs) are often equipped with multi-sample decoding strategies. Traditional text generation methods, such as beam search and sampling-based techniques, have notable limitations. This study explores the potential of arithmetic sampling, contrasting it with ancestral sampling.
  arXiv  Detail & Related papers  (2024-11-09T18:55:04Z)
• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued tokens based language modeling approach for text to speech synthesis (TTS) MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)
• Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation [49.827306773992376]
  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions. Our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-12-19T15:34:52Z)
• Sampling From Autoencoders' Latent Space via Quantization And Probability Mass Function Concepts [1.534667887016089]
  We introduce a novel post-training sampling algorithm rooted in the concept of probability mass functions, coupled with a quantization process. Our proposed algorithm establishes a vicinity around each latent vector from the input data and then proceeds to draw samples from these defined neighborhoods. This strategic approach ensures that the sampled latent vectors predominantly inhabit high-probability regions, which, in turn, can be effectively transformed into authentic real-world images.
  arXiv  Detail & Related papers  (2023-08-21T13:18:12Z)
• Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling [7.357511266926065]
  Conditional sampling of variational autoencoders (VAEs) is needed in various applications, such as missing data imputation, but is computationally intractable. A principled choice forally exact conditional sampling is Metropolis-within-Gibbs (MWG)
  arXiv  Detail & Related papers  (2023-08-17T16:08:18Z)
• Adaptive Siamese Tracking with a Compact Latent Network [219.38172719948048]
  We present an intuitive viewing to simplify the Siamese-based trackers by converting the tracking task to a classification. Under this viewing, we perform an in-depth analysis for them through visual simulations and real tracking examples. We apply it to adjust three classical Siamese-based trackers, namely SiamRPN++, SiamFC, and SiamBAN.
  arXiv  Detail & Related papers  (2023-02-02T08:06:02Z)
• Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning [57.163525407022966]
  Imbalanced learning is a fundamental challenge in data mining, where there is a disproportionate ratio of training samples in each class. Over-sampling is an effective technique to tackle imbalanced learning through generating synthetic samples for the minority class. We propose AutoSMOTE, an automated over-sampling algorithm that can jointly optimize different levels of decisions.
  arXiv  Detail & Related papers  (2022-08-26T04:28:01Z)
• Hardness of Samples Is All You Need: Protecting Deep Learning Models Using Hardness of Samples [1.2074552857379273]
  We show that the hardness degree of model extraction attacks samples is distinguishable from the hardness degree of normal samples. We propose Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks.
  arXiv  Detail & Related papers  (2021-06-21T22:03:31Z)
• iNNformant: Boundary Samples as Telltale Watermarks [68.8204255655161]
  We show that it is possible to generate sets of boundary samples which can identify any of four tested microarchitectures. These sets can be built to not contain any sample with a worse peak signal-to-noise ratio than 70dB.
  arXiv  Detail & Related papers  (2021-06-14T11:18:32Z)
• Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior [53.69310441063162]
  This paper proposes a sequential prior in a discrete latent space which can generate more naturally sounding samples. We evaluate the approach using listening tests, objective metrics of automatic speech recognition (ASR) performance, and measurements of prosody attributes.
  arXiv  Detail & Related papers  (2020-02-06T12:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.