MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing

• URL: http://arxiv.org/abs/2507.13401v1
• Date: Wed, 16 Jul 2025 20:56:57 GMT
• Title: MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing
• Authors: Shreya Kadambi, Risheek Garrepalli, Shubhankar Borse, Munawar Hyatt, Fatih Porikli,
• Abstract summary: Masking-Augmented Diffusion with Inference-Time Scaling (MADI) is a framework that improves the editability, compositionality and controllability of diffusion models.<n>First, we introduce Masking-Augmented gaussian Diffusion (MAgD), a novel training strategy with dual corruption process.<n>Second, we introduce an inference-time capacity scaling mechanism based on Pause Tokens, which act as special placeholders inserted into the prompt for increasing computational capacity at inference time.
• Score: 41.6713004141353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the remarkable success of diffusion models in text-to-image generation, their effectiveness in grounded visual editing and compositional control remains challenging. Motivated by advances in self-supervised learning and in-context generative modeling, we propose a series of simple yet powerful design choices that significantly enhance diffusion model capacity for structured, controllable generation and editing. We introduce Masking-Augmented Diffusion with Inference-Time Scaling (MADI), a framework that improves the editability, compositionality and controllability of diffusion models through two core innovations. First, we introduce Masking-Augmented gaussian Diffusion (MAgD), a novel training strategy with dual corruption process which combines standard denoising score matching and masked reconstruction by masking noisy input from forward process. MAgD encourages the model to learn discriminative and compositional visual representations, thus enabling localized and structure-aware editing. Second, we introduce an inference-time capacity scaling mechanism based on Pause Tokens, which act as special placeholders inserted into the prompt for increasing computational capacity at inference time. Our findings show that adopting expressive and dense prompts during training further enhances performance, particularly for MAgD. Together, these contributions in MADI substantially enhance the editability of diffusion models, paving the way toward their integration into more general-purpose, in-context generative diffusion architectures.

Related papers

• Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling [48.96034602889216]
  Variencoding Discrete Diffusion (VADD) is a novel framework that enhances discrete diffusion with latent variable modeling.<n>By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds and amortized inference over the training set.<n> Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines.
  arXiv  Detail & Related papers  (2025-05-23T01:45:47Z)
• Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory [33.78620829249978]
  Text-to-image (T2I) diffusion models have revolutionized generative modeling by producing high-fidelity, diverse, and visually realistic images. Recent attention-based methods have improved object inclusion and linguistic binding, but still face challenges such as attribute misbinding. We propose a Bayesian approach that designs custom priors over attention distributions to enforce desirable properties. Our approach treats the attention mechanism as an interpretable component, enabling fine-grained control and improved attribute-object alignment.
  arXiv  Detail & Related papers  (2024-11-25T10:57:48Z)
• DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models [79.0135981840682]
  We introduce DICE (Discrete Inversion for Controllable Editing), the first approach to enable precise inversion for discrete diffusion models.<n>By recording noise sequences and masking patterns during the reverse diffusion process, DICE enables accurate reconstruction and flexible editing of discrete data.<n>Our results show that DICE preserves high data fidelity while enhancing editing capabilities, offering new opportunities for fine-grained content manipulation in discrete spaces.
  arXiv  Detail & Related papers  (2024-10-10T17:59:48Z)
• Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models [4.703252654452953]
  This paper introduces a novel training-free algorithm in fine-grained generation, Aggregation of Multiple Diffusion Models (AMDM)<n>AMDM integrates features from multiple diffusion models into a specified model to activate specific features and enable fine-grained control.<n> Experimental results demonstrate that AMDM significantly improves fine-grained control without training, validating its effectiveness.
  arXiv  Detail & Related papers  (2024-10-02T06:16:06Z)
• Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors [56.82596340418697]
  We propose a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.
  arXiv  Detail & Related papers  (2024-01-29T10:36:57Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• Collaborative Diffusion for Multi-Modal Face Generation and Editing [34.16906110777047]
  We present Collaborative Diffusion, where pre-trained uni-modal diffusion models collaborate to achieve multi-modal face generation and editing without re-training. Specifically, we propose dynamic diffuser, a meta-network that adaptively hallucinates multi-modal denoising steps by predicting the spatial-temporal influence functions for each pre-trained uni-modal model.
  arXiv  Detail & Related papers  (2023-04-20T17:59:02Z)
• Diffusion Models as Masked Autoencoders [52.442717717898056]
  We revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE) We perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.
  arXiv  Detail & Related papers  (2023-04-06T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.