Show-o Turbo: Towards Accelerated Unified Multimodal Understanding and Generation

• URL: http://arxiv.org/abs/2502.05415v1
• Date: Sat, 08 Feb 2025 02:52:25 GMT
• Title: Show-o Turbo: Towards Accelerated Unified Multimodal Understanding and Generation
• Authors: Chenkai Xu, Xu Wang, Zhenyi Liao, Yishun Li, Tianqi Hou, Zhijie Deng,
• Abstract summary: Show-o is a multimodal understanding model for text-to-image and image-to-text generation.<n>This paper introduces Show-o Turbo to bridge the gap between Show-o and other approaches.<n>Show-o Turbo exhibits a 1.5x speedup without significantly sacrificing performance.
• Score: 12.34529497235534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There has been increasing research interest in building unified multimodal understanding and generation models, among which Show-o stands as a notable representative, demonstrating great promise for both text-to-image and image-to-text generation. The inference of Show-o involves progressively denoising image tokens and autoregressively decoding text tokens, and hence, unfortunately, suffers from inefficiency issues from both sides. This paper introduces Show-o Turbo to bridge the gap. We first identify a unified denoising perspective for the generation of images and text in Show-o based on the parallel decoding of text tokens. We then propose to extend consistency distillation (CD), a qualified approach for shortening the denoising process of diffusion models, to the multimodal denoising trajectories of Show-o. We introduce a trajectory segmentation strategy and a curriculum learning procedure to improve the training convergence. Empirically, in text-to-image generation, Show-o Turbo displays a GenEval score of 0.625 at 4 sampling steps without using classifier-free guidance (CFG), outperforming that of the original Show-o with 8 steps and CFG; in image-to-text generation, Show-o Turbo exhibits a 1.5x speedup without significantly sacrificing performance. The code is available at https://github.com/zhijie-group/Show-o-Turbo.

Related papers

• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer. Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis. In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• Unified Autoregressive Visual Generation and Understanding with Continuous Tokens [52.21981295470491]
  We present UniFluid, a unified autoregressive framework for joint visual generation and understanding. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other.
  arXiv  Detail & Related papers  (2025-03-17T17:58:30Z)
• Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think [38.258453761376586]
  We propose Dream Engine, an efficient framework designed for arbitrary text-image interleaved control in image generation models. Our approach utilizes a two-stage training paradigm, consisting of joint text-image alignment and multimodal interleaved instruction tuning. Our experiments demonstrate that this training method is effective, achieving a 0.69 overall score on the GenEval benchmark.
  arXiv  Detail & Related papers  (2025-02-27T15:08:39Z)
• DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation [46.5013105017258]
  Diffusion models are trained by denoising a Markovian process which gradually adds noise to the input.<n>We propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework.
  arXiv  Detail & Related papers  (2024-10-10T17:41:54Z)
• FIFO-Diffusion: Generating Infinite Videos from Text without Training [44.65468310143439]
  FIFO-Diffusion is conceptually capable of generating infinitely long videos without additional training. Our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines.
  arXiv  Detail & Related papers  (2024-05-19T07:48:41Z)
• TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation [44.740794326596664]
  TheaterGen is a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models. Within this framework, LLMs, acting as "Screenwriter", engage in multi-turn interaction, generating and managing a standardized prompt book. With the effective management of prompt books and character images, TheaterGen significantly improves semantic and contextual consistency in synthesized images.
  arXiv  Detail & Related papers  (2024-04-29T17:58:14Z)
• Learning from Mistakes: Iterative Prompt Relabeling for Text-to-Image Diffusion Model Training [33.51524424536508]
  Iterative Prompt Relabeling (IPR) is a novel algorithm that aligns images to text through iterative image sampling and prompt relabeling with feedback. We conduct thorough experiments on SDv2 and SDXL, testing their capability to follow instructions on spatial relations.
  arXiv  Detail & Related papers  (2023-12-23T11:10:43Z)
• AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation [138.98095392584693]
  We introduce Auto-Regressive Diffusion (AR-Diffusion) to account for the inherent sequential characteristic of natural language. AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps. In a series of experiments on various text generation tasks, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models.
  arXiv  Detail & Related papers  (2023-05-16T15:10:22Z)
• Shifted Diffusion for Text-to-image Generation [65.53758187995744]
  Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input text. Corgi also achieves new state-of-the-art results across different datasets on downstream language-free text-to-image generation tasks.
  arXiv  Detail & Related papers  (2022-11-24T03:25:04Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation [52.40490994871753]
  We introduce a contrastive loss to representations BEfore Fusing (ALBEF) through cross-modal attention. We propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks.
  arXiv  Detail & Related papers  (2021-07-16T00:19:22Z)
• TIME: Text and Image Mutual-Translation Adversarial Networks [55.1298552773457]
  We propose Text and Image Mutual-Translation Adversarial Networks (TIME) TIME learns a T2I generator G and an image captioning discriminator D under the Generative Adversarial Network framework. In experiments, TIME achieves state-of-the-art (SOTA) performance on the CUB and MS-COCO dataset.
  arXiv  Detail & Related papers  (2020-05-27T06:40:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.