Tractable Transformers for Flexible Conditional Generation

• URL: http://arxiv.org/abs/2502.07616v1
• Date: Tue, 11 Feb 2025 15:05:26 GMT
• Title: Tractable Transformers for Flexible Conditional Generation
• Authors: Anji Liu, Xuejie Liu, Dayuan Zhao, Mathias Niepert, Yitao Liang, Guy Van den Broeck,
• Abstract summary: Tractable Transformers (Tracformer) is a Transformer-based generative model that is more robust to different conditional generation tasks. This paper proposes Tractable Transformers, a Transformer-based generative model that is more robust to different conditional generation tasks.
• Score: 44.52426555357705
• License:
• Abstract: Non-autoregressive (NAR) generative models are valuable because they can handle diverse conditional generation tasks in a more principled way than their autoregressive (AR) counterparts, which are constrained by sequential dependency requirements. Recent advancements in NAR models, such as diffusion language models, have demonstrated superior performance in unconditional generation compared to AR models (e.g., GPTs) of similar sizes. However, such improvements do not always lead to improved conditional generation performance. We show that a key reason for this gap is the difficulty in generalizing to conditional probability queries unseen during training. As a result, strong unconditional generation performance does not guarantee high-quality conditional generation. This paper proposes Tractable Transformers (Tracformer), a Transformer-based generative model that is more robust to different conditional generation tasks. Unlike existing models that rely solely on global contextual features derived from full inputs, Tracformers incorporate a sparse Transformer encoder to capture both local and global contextual information. This information is routed through a decoder for conditional generation. Empirical results demonstrate that Tracformers achieve state-of-the-art conditional generation performance on text modeling compared to recent diffusion and AR model baselines.

Related papers

• Adaptively Controllable Diffusion Model for Efficient Conditional Image Generation [8.857237929151795]
  We propose a new adaptive framework, $textitAdaptively Controllable Diffusion (AC-Diff) Model$, to automatically and fully control the generation process. AC-Diff is expected to largely reduce the average number of generation steps and execution time while maintaining the same performance as done in the literature diffusion models.
  arXiv  Detail & Related papers  (2024-11-19T21:26:30Z)
• Conditional [MASK] Discrete Diffusion Language Model [14.208510167132983]
  Diffusion-EAGS is a framework that integrates conditional masked language models into diffusion language models. We show that Diffusion-EAGS achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.
  arXiv  Detail & Related papers  (2024-11-10T11:49:36Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• DiffusER: Discrete Diffusion via Edit-based Reconstruction [88.62707047517914]
  DiffusER is an edit-based generative model for text based on denoising diffusion models. It can rival autoregressive models on several tasks spanning machine translation, summarization, and style transfer. It can also perform other varieties of generation that standard autoregressive models are not well-suited for.
  arXiv  Detail & Related papers  (2022-10-30T16:55:23Z)
• Conditional Generation with a Question-Answering Blueprint [84.95981645040281]
  We advocate planning as a useful intermediate representation for rendering conditional generation less opaque and more grounded. We obtain blueprints automatically by exploiting state-of-the-art question generation technology. We develop Transformer-based models, each varying in how they incorporate the blueprint in the generated output.
  arXiv  Detail & Related papers  (2022-07-01T13:10:19Z)
• Regression Transformer: Concurrent Conditional Generation and Regression by Blending Numerical and Textual Tokens [3.421506449201873]
  The Regression Transformer (RT) casts continuous properties as sequences of numerical tokens and encodes them jointly with conventional tokens. We propose several extensions to the XLNet objective and adopt an alternating training scheme to concurrently optimize property prediction and conditional text generation. This finds application particularly in property-driven, local exploration of the chemical or protein space.
  arXiv  Detail & Related papers  (2022-02-01T08:57:31Z)
• Controllable and Compositional Generation with Latent-Space Energy-Based Models [60.87740144816278]
  Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. By composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.
  arXiv  Detail & Related papers  (2021-10-21T03:31:45Z)
• Non-autoregressive Transformer-based End-to-end ASR using BERT [13.07939371864781]
  This paper presents a transformer-based end-to-end automatic speech recognition (ASR) model based on BERT. A series of experiments conducted on the AISHELL-1 dataset demonstrates competitive or superior results.
  arXiv  Detail & Related papers  (2021-04-10T16:22:17Z)
• Non-Autoregressive Translation by Learning Target Categorical Codes [59.840510037250944]
  We propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. Experiment results show that our model achieves comparable or better performance in machine translation tasks.
  arXiv  Detail & Related papers  (2021-03-21T14:12:34Z)
• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation [39.577220559911055]
  We investigate large-scale latent variable models (LVMs) for neural story generation with objectives in two threads: generation effectiveness and controllability. We advocate to revive latent variable modeling, essentially the power of representation learning, in the era of Transformers. Specifically, we integrate latent representation vectors with a Transformer-based pre-trained architecture to build conditional variational autoencoder (CVAE)
  arXiv  Detail & Related papers  (2021-01-04T08:31:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.