CART: Compositional Auto-Regressive Transformer for Image Generation

• URL: http://arxiv.org/abs/2411.10180v2
• Date: Mon, 29 Sep 2025 17:57:08 GMT
• Title: CART: Compositional Auto-Regressive Transformer for Image Generation
• Abstract summary: We propose a novel Auto-Regressive (AR) image generation approach that models images as hierarchical compositions of interpretable visual layers.<n>CART adds image details iteratively via semantically meaningful decompositions.<n>We show CART generates visually compelling results while enabling structured image manipulation.
• Score: 2.48490797934472
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel Auto-Regressive (AR) image generation approach that models images as hierarchical compositions of interpretable visual layers. While AR models have achieved transformative success in language modeling, replicating this success in vision tasks has presented unique challenges due to inherent spatial dependencies in images. Addressing the unique challenges of vision tasks, our method (CART) adds image details iteratively via semantically meaningful decompositions. We demonstrate the flexibility and generality of CART by applying it across three distinct decomposition strategies: (i) Base-Detail Decomposition (Mumford-Shah smoothness), (ii) Intrinsic Decomposition (albedo/shading), and (iii) Specularity Decomposition (diffuse/specular). This "next-detail" strategy outperforms traditional "next-token" and "next-scale" approaches, improving controllability, semantic interpretability, and resolution scalability. Experiments show CART generates visually compelling results while enabling structured image manipulation, opening new directions for controllable generative modeling via physically or perceptually motivated image factorization.

Related papers

• Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information [10.086588302174263]
  We propose a principled approach, termed deep modality-shared self-expressive model (DeepMORSE)<n>DeepMORSE discovers cross-modal structures via a modality-shared self-expressive model and simultaneously learns structured representations that conform to a union of modality-specific subspaces.<n>We evaluate our DeepMORSE on six widely used image clustering benchmarks and observe performance improvements exceeding 3%.
  arXiv  Detail & Related papers  (2026-08-09T02:23:49Z)
• DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable [10.070492917829995]
  We introduce DrawAI, comprising an agentic benchmark, DrawAI-Bench, and a reconstruction workflow, DrawAI-Flow.<n>We formulate image-to-editable reconstruction, which recovers a structured, directly manipulable artifact from an image while preserving its visual and semantic content.<n>The results show that reconstruction quality and costs vary substantially across model-harness configurations, while DrawAI-Flow consistently improves editable structure.
  arXiv  Detail & Related papers  (2026-08-01T09:05:06Z)
• StyleVAR: Controllable Image Style Transfer via Visual Autoregressive Modeling [0.0]
  We formulate style transfer as conditional discrete sequence modeling in a learned latent space.<n>We introduce a blended cross-attention mechanism in which the evolving target representation attends to its own history.<n>Style VAR consistently outperforms an AdaIN baseline on Style Loss, Content Loss, LPIPS, SSIM, DreamSim, and CLIP similarity.
  arXiv  Detail & Related papers  (2026-04-22T19:52:35Z)
• CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval [54.15776146365823]
  Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text.<n>We propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components.
  arXiv  Detail & Related papers  (2026-01-07T09:21:38Z)
• Image-POSER: Reflective RL for Multi-Expert Image Generation and Editing [16.943575863059607]
  Image-POSER orchestrates a diverse registry of pretrained text-to-image and image-to-image experts.<n>It handles long-form prompts end-to-end through dynamic task decomposition.<n>It is consistently preferred in human evaluations.
  arXiv  Detail & Related papers  (2025-11-15T03:15:34Z)
• BLIP3o-NEXT: Next Frontier of Native Image Generation [113.25832679864631]
  We present BLIP3o, a fully open foundation model in the BLIP3 series that advances the next frontier of native image generation.<n>BLIP3o unifies text-to-image generation and image editing within a single architecture, demonstrating strong image generation and image editing capabilities.
  arXiv  Detail & Related papers  (2025-10-17T17:50:58Z)
• Policy Optimized Text-to-Image Pipeline Design [72.87655664038617]
  We introduce a novel reinforcement learning-based framework for text-to-image generation.<n>Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations.<n>We then implement a two-phase training strategy: initial vocabulary training followed by GRPO-based optimization.
  arXiv  Detail & Related papers  (2025-05-27T17:50:47Z)
• Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation [54.588082888166504]
  We present Mogao, a unified framework that enables interleaved multi-modal generation through a causal approach.<n>Mogoo integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance.<n>Experiments show that Mogao achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs.
  arXiv  Detail & Related papers  (2025-05-08T17:58:57Z)
• Boosting Generative Image Modeling via Joint Image-Feature Synthesis [10.32324138962724]
  We introduce a novel generative image modeling framework that seamlessly bridges the gap by leveraging a diffusion model to jointly model low-level image latents. Our latent-semantic diffusion approach learns to generate coherent image-feature pairs from pure noise. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance.
  arXiv  Detail & Related papers  (2025-04-22T17:41:42Z)
• Visual Autoregressive Modeling for Image Super-Resolution [14.935662351654601]
  We propose a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction. We collect large-scale data and design a training process to obtain robust generative priors.
  arXiv  Detail & Related papers  (2025-01-31T09:53:47Z)
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step [77.86514804787622]
  Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. We provide the first comprehensive investigation of the potential of CoT reasoning to enhance autoregressive image generation. We propose the Potential Assessment Reward Model (PARM) and PARM++, specialized for autoregressive image generation.
  arXiv  Detail & Related papers  (2025-01-23T18:59:43Z)
• Hierarchical Information Flow for Generalized Efficient Image Restoration [108.83750852785582]
  We propose a hierarchical information flow mechanism for image restoration, dubbed Hi-IR.<n>Hi-IR constructs a hierarchical information tree representing the degraded image across three levels.<n>In seven common image restoration tasks, Hi-IR achieves its effectiveness and generalizability.
  arXiv  Detail & Related papers  (2024-11-27T18:30:08Z)
• Reward Incremental Learning in Text-to-Image Generation [26.64026346266299]
  We present Reward Incremental Distillation (RID), a method that mitigates forgetting with minimal computational overhead. The experimental results demonstrate the efficacy of RID in achieving consistent, high-quality gradient generation in RIL scenarios.
  arXiv  Detail & Related papers  (2024-11-26T10:54:33Z)
• High-Resolution Image Synthesis via Next-Token Prediction [19.97037318862443]
  We introduce textbfD-JEPA$cdot$T2I, an autoregressive model based on continuous tokens to generate high-quality, photorealistic images at arbitrary resolutions, up to 4K. For the first time, we achieve state-of-the-art high-resolution image synthesis via next-token prediction.
  arXiv  Detail & Related papers  (2024-11-22T09:08:58Z)
• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling [64.09238330331195]
  We propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss. We show that MMAR demonstrates much more superior performance than other joint multi-modal models.
  arXiv  Detail & Related papers  (2024-10-14T17:57:18Z)
• Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis [62.06970466554273]
  We present Meissonic, which non-autoregressive masked image modeling (MIM) text-to-image elevates to a level comparable with state-of-the-art diffusion models like SDXL. We leverage high-quality training data, integrate micro-conditions informed by human preference scores, and employ feature compression layers to further enhance image fidelity and resolution. Our model not only matches but often exceeds the performance of existing models like SDXL in generating high-quality, high-resolution images.
  arXiv  Detail & Related papers  (2024-10-10T17:59:17Z)
• A Spitting Image: Modular Superpixel Tokenization in Vision Transformers [0.0]
  Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction.
  arXiv  Detail & Related papers  (2024-08-14T17:28:58Z)
• ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation [7.645341879105626]
  We present Blur, a novel curriculum learning approach to improve layout-to-image generation models. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images.
  arXiv  Detail & Related papers  (2024-04-11T08:50:12Z)
• FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis [48.9652334528436]
  We introduce an innovative, training-free approach FouriScale from the perspective of frequency domain analysis. We replace the original convolutional layers in pre-trained diffusion models by incorporating a dilation technique along with a low-pass operation. Our method successfully balances the structural integrity and fidelity of generated images, achieving an astonishing capacity of arbitrary-size, high-resolution, and high-quality generation.
  arXiv  Detail & Related papers  (2024-03-19T17:59:33Z)
• CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation [49.3016007471979]
  Large generative diffusion models have revolutionized text-to-image generation and offer immense potential for conditional generation tasks. However, their widespread adoption is hindered by the high computational cost, which limits their real-time application. We introduce a novel method dubbed CoDi, that adapts a pre-trained latent diffusion model to accept additional image conditioning inputs.
  arXiv  Detail & Related papers  (2023-10-02T17:59:18Z)
• StraIT: Non-autoregressive Generation with Stratified Image Transformer [63.158996766036736]
  Stratified Image Transformer(StraIT) is a pure non-autoregressive(NAR) generative model. Our experiments demonstrate that StraIT significantly improves NAR generation and out-performs existing DMs and AR methods.
  arXiv  Detail & Related papers  (2023-03-01T18:59:33Z)
• A Generic Approach for Enhancing GANs by Regularized Latent Optimization [79.00740660219256]
  We introduce a generic framework called em generative-model inference that is capable of enhancing pre-trained GANs effectively and seamlessly. Our basic idea is to efficiently infer the optimal latent distribution for the given requirements using Wasserstein gradient flow techniques.
  arXiv  Detail & Related papers  (2021-12-07T05:22:50Z)
• BIGRoC: Boosting Image Generation via a Robust Classifier [27.66648389933265]
  We propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images. Our method, termed BIGRoC, is based on a post-processing procedure via the guidance of a given robust classifier.
  arXiv  Detail & Related papers  (2021-08-08T18:05:44Z)
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [74.29384873537587]
  We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images.
  arXiv  Detail & Related papers  (2021-03-18T05:10:49Z)
• Learning Enriched Features for Real Image Restoration and Enhancement [166.17296369600774]
  convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. We present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
  arXiv  Detail & Related papers  (2020-03-15T11:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.