MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation

• URL: http://arxiv.org/abs/2608.05450v1
• Date: Wed, 05 Aug 2026 22:52:41 GMT
• Title: MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
• Abstract summary: MOSAIK is a damage-guided framework that varies patch size across regions and denoising steps.<n>Our approach delivers highly competitive performance at moderate budgets.
• Score: 26.941280093248263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pixel-space diffusion models avoid the reconstruction ceiling of latent diffusion models by generating directly in image space. However, their substantially higher token count makes generation expensive due to the quadratic complexity of self-attention. Several existing efficiency methods reduce this cost by using larger patches at selected denoising steps, thereby representing the image with fewer tokens. Yet, each step still uses a single patch size uniformly across the entire image, overlooking that different regions suffer different fidelity losses when coarsened. We introduce MOSAIK, a damage-guided framework that varies patch size across regions and denoising steps. MOSAIK adapts the PixelDiT backbone to generate arbitrary heterogeneous patch layouts, and a lightweight predictor uses intermediate denoising features to estimate the fidelity loss caused by coarsening each region. Given a token budget, our damage-guided layout predictor assigns fine patches to sensitive regions and coarse patches elsewhere. Remarkably, while reducing FLOPs by 70% and token count by 83%, MOSAIK matches the full-compute PixelDiT on GenEval and its DPG-Bench score drops by only 1.0 point. Compared to diverse efficiency paradigms, including temporal patch scheduling and feature caching, our approach delivers highly competitive performance at moderate budgets and consistently outperforms these baselines in highly constrained compute regimes.

Related papers

• P-PatchDiff: Progressive Patch Diffusion Models for Low-light Image Enhancement [16.721528049675424]
  We propose P-PatchDiff, a scalable progressive patch diffusion framework for low-light image enhancement.<n>It dynamically adjusts patch size throughout the denoising process, enabling a gradual shift from local to global views.<n>We empirically demonstrate that P-PatchDiff effectively enhances images ranging from 400 $times$ 600 to 4K.
  arXiv  Detail & Related papers  (2026-09-01T12:01:17Z)
• DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation [10.719563134726057]
  We present DPAR, a decoder-only autoregressive model that aggregates image tokens into a variable number of patches for efficient image generation.<n>DPAR reduces token count by 1.81x and 2.06x on Imagenet 256 and 384 generation resolution respectively, leading to a reduction of up to 40% FLOPs in training costs.
  arXiv  Detail & Related papers  (2025-12-26T05:03:47Z)
• DiP: Taming Diffusion Models in Pixel Space [91.51011771517683]
  Diffusion Transformer (DiT) backbone operates on large patches for efficient global structure construction.<n>Co-trained lightweight Patch Detailer Head leverages contextual features to restore fine-grained local details.
  arXiv  Detail & Related papers  (2025-11-24T06:55:49Z)
• Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesis [40.93077975823353]
  Visual autoregressive modeling, based on the next-scale prediction paradigm, exhibits notable advantages in image quality and model scalability.<n>However, the computational overhead in high-resolution stages remains a critical challenge due to the substantial number of tokens involved.<n>We introduce Sparsevar, a plug-and-play acceleration framework for next-scale prediction that dynamically excludes low-frequency tokens during inference without requiring additional training.
  arXiv  Detail & Related papers  (2025-07-28T01:13:24Z)
• Robust Representation Consistency Model via Contrastive Denoising [83.47584074390842]
  randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations.<n> diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples.<n>We reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space.
  arXiv  Detail & Related papers  (2025-01-22T18:52:06Z)
• DiffPAD: Denoising Diffusion-based Adversarial Patch Decontamination [5.7254228484416325]
  DiffPAD is a novel framework that harnesses the power of diffusion models for adversarial patch decontamination. We show that DiffPAD achieves state-of-the-art adversarial robustness against patch attacks and excels in recovering naturalistic images without patch remnants.
  arXiv  Detail & Related papers  (2024-10-31T15:09:36Z)
• Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training [61.61368146268329]
  We propose Pixel-reweighted AdveRsarial Training (PART), a new framework that partially reduces $epsilon$ for less influential pixels. Part achieves a notable improvement in accuracy without compromising robustness on CIFAR-10, SVHN and TinyImagenet-200.
  arXiv  Detail & Related papers  (2024-06-02T09:43:34Z)
• PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution [44.345740602726345]
  PatchScaler is an efficient patch-independent diffusion pipeline for single image super-resolution. A texture prompt adaptively retrieves texture priors for the target patch from a common reference texture memory. Our code achieves superior performance in both quantitative and qualitative evaluations, while significantly speeding up inference.
  arXiv  Detail & Related papers  (2024-05-27T13:31:46Z)
• Learning to Rank Patches for Unbiased Image Redundancy Reduction [80.93989115541966]
  Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. We propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches.
  arXiv  Detail & Related papers  (2024-03-31T13:12:41Z)
• SA-MixNet: Structure-aware Mixup and Invariance Learning for Scribble-supervised Road Extraction in Remote Sensing Images [85.52629779976137]
  We propose a structure-aware Mixup scheme to paste road regions from one image onto another for creating an image scene with increased complexity. A discriminator-based regularization is designed for enhancing the connectivity meanwhile preserving the structure of roads. Our framework demonstrates superior performance on the DeepGlobe, Wuhan, and Massachusetts datasets.
  arXiv  Detail & Related papers  (2024-03-03T02:56:43Z)
• Cross-domain Self-supervised Framework for Photoacoustic Computed Tomography Image Reconstruction [4.769412124596113]
  We propose a cross-domain unsupervised reconstruction (CDUR) strategy with a pure transformer model. We implement a self-supervised reconstruction in a model-based form and leverage the self-supervision to enforce the measurement and image consistency. Experimental results on in-vivo PACT dataset of mice demonstrate the potential of our unsupervised framework.
  arXiv  Detail & Related papers  (2023-01-17T03:47:01Z)
• Image Fine-grained Inpainting [89.17316318927621]
  We present a one-stage model that utilizes dense combinations of dilated convolutions to obtain larger and more effective receptive fields. To better train this efficient generator, except for frequently-used VGG feature matching loss, we design a novel self-guided regression loss. We also employ a discriminator with local and global branches to ensure local-global contents consistency.
  arXiv  Detail & Related papers  (2020-02-07T03:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.