Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers

• URL: http://arxiv.org/abs/2603.27666v1
• Date: Sun, 29 Mar 2026 12:31:34 GMT
• Title: Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers
• Abstract summary: This paper explores controllable diffusion models built upon linear attention architectures.<n>We propose a novel controllable diffusion framework tailored for linear attention backbones like SANA.<n>Our approach achieves state-of-the-art controllable generation performance based on linear-attention models.
• Score: 69.60693631859411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in diffusion-based controllable visual generation have led to remarkable improvements in image quality. However, these powerful models are typically deployed on cloud servers due to their large computational demands, raising serious concerns about user data privacy. To enable secure and efficient on-device generation, we explore in this paper controllable diffusion models built upon linear attention architectures, which offer superior scalability and efficiency, even on edge devices. Yet, our experiments reveal that existing controllable generation frameworks, such as ControlNet and OminiControl, either lack the flexibility to support multiple heterogeneous condition types or suffer from slow convergence on such linear-attention models. To address these limitations, we propose a novel controllable diffusion framework tailored for linear attention backbones like SANA. The core of our method lies in a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs, such as spatially aligned and non-aligned cues. Extensive experiments on multiple tasks and benchmarks demonstrate that our approach achieves state-of-the-art controllable generation performance based on linear-attention models, surpassing existing methods in terms of fidelity and controllability.

Related papers

• From Open Loop to Closed Loop: A Test-Time Iterative Optimization Framework for Reference-Consistent Image Generation [108.02425846269234]
  controllable image generation has made significant strides by incorporating visual reference conditions.<n>Existing methods predominantly operate as open-loop systems.<n>We propose a novel test-time iterative optimization framework that reformulates reference-consistent generation as a closed-loop dynamic tracking problem.
  arXiv  Detail & Related papers  (2026-07-06T05:36:30Z)
• Drift Flow Matching [69.16550765154163]
  We propose Drift Flow Matching, a framework that connects drifting generative modeling with flow-based iterative generation.<n>DFM preserves the efficiency of direct transport maps while enabling generation to be refined through multiple inference steps.
  arXiv  Detail & Related papers  (2026-05-17T03:58:01Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation [75.58351043849385]
  Generative models often treat continuous data and discrete events as separate processes, creating a gap in modeling complex systems where they unify synchronously.<n>To bridge this gap, we introduce JointDiff, a novel diffusion framework designed to interact these two processes by simultaneously generating continuous-temporal data and synchronous discrete events.<n>JointDiff achieves state-of-the-art performance, demonstrating that joint modeling is crucial for building realistic and controllable models for interactive systems.
  arXiv  Detail & Related papers  (2025-09-26T16:04:00Z)
• SCALAR: Scale-wise Controllable Visual Autoregressive Learning [15.775596699630633]
  We present SCALAR, a controllable generation method based on Visual Autoregressive ( VAR)<n>We leverage a pretrained image encoder to extract semantic control signal encodings, which are projected into scale-specific representations and injected into the corresponding layers of the VAR backbone.<n>Building on SCALAR, we develop SCALAR-Uni, a unified extension that aligns multiple control modalities into a shared latent space, supporting flexible multi-conditional guidance in a single model.
  arXiv  Detail & Related papers  (2025-07-26T13:23:08Z)
• A Practical Investigation of Spatially-Controlled Image Generation with Transformers [23.373853892284227]
  We aim to provide clear takeaways across generation paradigms for spatially-controlled generation.<n>We perform experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models.<n>We re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data.
  arXiv  Detail & Related papers  (2025-07-21T15:33:49Z)
• Controllable Motion Generation via Diffusion Modal Coupling [19.534234002173314]
  We propose a novel framework that enhances controllability in diffusion models by leveraging multi-modal prior distributions.<n>We evaluate our approach on motion prediction using a dataset and multi-task control in Maze2D environments.
  arXiv  Detail & Related papers  (2025-03-04T07:22:34Z)
• CAR: Controllable Autoregressive Modeling for Visual Generation [100.33455832783416]
  Controllable AutoRegressive Modeling (CAR) is a novel, plug-and-play framework that integrates conditional control into multi-scale latent variable modeling. CAR progressively refines and captures control representations, which are injected into each autoregressive step of the pre-trained model to guide the generation process. Our approach demonstrates excellent controllability across various types of conditions and delivers higher image quality compared to previous methods.
  arXiv  Detail & Related papers  (2024-10-07T00:55:42Z)
• ControlVAR: Exploring Controllable Visual Autoregressive Modeling [48.66209303617063]
  Conditional visual generation has witnessed remarkable progress with the advent of diffusion models (DMs) Challenges such as expensive computational cost, high inference latency, and difficulties of integration with large language models (LLMs) have necessitated exploring alternatives to DMs. This paper introduces Controlmore, a novel framework that explores pixel-level controls in visual autoregressive modeling for flexible and efficient conditional generation.
  arXiv  Detail & Related papers  (2024-06-14T06:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.