Controllable Image Generation with Composed Parallel Token Prediction

• URL: http://arxiv.org/abs/2604.05730v1
• Date: Tue, 07 Apr 2026 11:33:52 GMT
• Title: Controllable Image Generation with Composed Parallel Token Prediction
• Abstract summary: Conditional discrete generative models struggle to faithfully compose multiple input conditions.<n>We derive a theoretically-grounded formulation for composing discrete probabilistic generative processes.<n>Our method offers a $2.3times$ to $12times$ real-time speed-up over comparable methods.
• Score: 26.82273247382564
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a $63.4\%$ relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of $-9.58$. Meanwhile, our method offers a $2.3\times$ to $12\times$ real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.

Related papers

• DiffusionGemma Technical Report [72.10570774996219]
  We introduce DiffusionGemma, a language model that uses discrete diffusion to generate text at exceptionally high speed.<n>Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel.<n>Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget.
  arXiv  Detail & Related papers  (2026-07-31T16:11:46Z)
• MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis [46.56871839630665]
  MixDiffusion is a training-free diffusion framework for multi-condition T2I generation.<n>It theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text.
  arXiv  Detail & Related papers  (2026-07-20T07:40:56Z)
• Score Distillation of Flow Matching Models [67.86066177182046]
  We extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models.<n>SiD works out of the box across these models, in both data-free and data-aided settings.<n>This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models.
  arXiv  Detail & Related papers  (2025-09-29T17:45:48Z)
• Discrete Diffusion Models for Language Generation [0.0]
  This thesis investigates the feasibility and performance of discrete diffusion models for natural language generation.<n>We use Bits Per Token (BPT), Negative Log-Likelihood (NLL), Perplexity (PPL), and Batch Processing Speed to assess generative performance.<n>The AR model outperforms in compression with a lower mean BPT of 4.59, but D3PM achieves higher processing speed, reaching up to 3.97 batches per sec.
  arXiv  Detail & Related papers  (2025-07-02T23:43:02Z)
• Cost-Aware Routing for Efficient Text-To-Image Generation [19.848723289971208]
  We propose a framework to allow the amount of computation to vary for each prompt, depending on its complexity.<n>We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine already-trained text-to-image models, our approach is able to deliver an average quality that is higher than that achievable by any of these models alone.
  arXiv  Detail & Related papers  (2025-06-17T17:48:50Z)
• Fast constrained sampling in pre-trained diffusion models [80.99262780028015]
  We propose an algorithm that enables fast, high-quality generation under arbitrary constraints.<n>Our approach produces results that rival or surpass the state-of-the-art training-free inference methods.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Schr\"odinger bridge based deep conditional generative learning [0.0]
  We introduce a novel Schr"odinger bridge based deep generative method for learning conditional distributions. We apply our method to both low-dimensional and high-dimensional conditional generation problems.
  arXiv  Detail & Related papers  (2024-09-25T19:08:13Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Controllable Image Generation With Composed Parallel Token Prediction [5.107886283951882]
  compositional image generation requires models to generalise well in situations where two or more input concepts do not necessarily appear together in training. We propose a formulation for controllable conditional generation of images via composing the log-probability outputs of discrete generative models of the latent space.
  arXiv  Detail & Related papers  (2024-05-10T15:27:35Z)
• AdaDiff: Adaptive Step Selection for Fast Diffusion Models [82.78899138400435]
  We introduce AdaDiff, a lightweight framework designed to learn instance-specific step usage policies.<n>AdaDiff is optimized using a policy method to maximize a carefully designed reward function.<n>We conduct experiments on three image generation and two video generation benchmarks and demonstrate that our approach achieves similar visual quality compared to the baseline.
  arXiv  Detail & Related papers  (2023-11-24T11:20:38Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.