EditAR: Unified Conditional Generation with Autoregressive Models

• URL: http://arxiv.org/abs/2501.04699v1
• Date: Wed, 08 Jan 2025 18:59:35 GMT
• Title: EditAR: Unified Conditional Generation with Autoregressive Models
• Authors: Jiteng Mu, Nuno Vasconcelos, Xiaolong Wang,
• Abstract summary: We propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods.
• Score: 58.093860528672735
• License:
• Abstract: Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: https://jitengmu.github.io/EditAR/

Related papers

• DreamOmni: Unified Image Generation and Editing [51.45871494724542]
  We introduce Dream Omni, a unified model for image generation and editing. For training, Dream Omni jointly trains T2I generation and downstream tasks. This collaboration significantly boosts editing performance.
  arXiv  Detail & Related papers  (2024-12-22T17:17:28Z)
• GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis [10.47359822447001]
  We present an alternate paradigm for T2I synthesis, decomposing the task of complex multi-step generation into three steps. Our approach derives its strength from the fact that it is modular in nature, is training free, and can be applied over any combination of image generation and editing models.
  arXiv  Detail & Related papers  (2024-12-08T22:29:56Z)
• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks. Our approach enables versatile capabilities via different inference-time sampling schemes. Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
  Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks. We provide an exhaustive overview of existing methods using diffusion models for image editing. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
  arXiv  Detail & Related papers  (2024-02-27T14:07:09Z)
• Instruct-Imagen: Image Generation with Multi-modal Instruction [90.04481955523514]
  instruct-imagen is a model that tackles heterogeneous image generation tasks and generalizes across unseen tasks. We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision. Human evaluation on various image generation datasets reveals that instruct-imagen matches or surpasses prior task-specific models in-domain.
  arXiv  Detail & Related papers  (2024-01-03T19:31:58Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• SINE: SINgle Image Editing with Text-to-Image Diffusion Models [10.67527134198167]
  This work aims to address the problem of single-image editing. We propose a novel model-based guidance built upon the classifier-free guidance. We show promising editing capabilities, including changing style, content addition, and object manipulation.
  arXiv  Detail & Related papers  (2022-12-08T18:57:13Z)
• Learning to Model Editing Processes [98.11448946134894]
  We propose modeling editing processes, modeling the whole process of iteratively generating sequences. We form a conceptual framework to describe the likelihood of multi-step edits, and describe neural models that can learn a generative model of sequences based on these multistep edits.
  arXiv  Detail & Related papers  (2022-05-24T21:32:52Z)
• EdiBERT, a generative model for image editing [12.605607949417033]
  EdiBERT is a bi-directional transformer trained in the discrete latent space built by a vector-quantized auto-encoder. We show that the resulting model matches state-of-the-art performances on a wide variety of tasks.
  arXiv  Detail & Related papers  (2021-11-30T10:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.