Linearizing Vision Transformer with Test-Time Training

• URL: http://arxiv.org/abs/2605.02772v1
• Date: Mon, 04 May 2026 16:16:26 GMT
• Title: Linearizing Vision Transformer with Test-Time Training
• Abstract summary: Inheriting weights from pretrained Transformers provides an appealing shortcut, yet the fundamental representational gap between Softmax and linear attention prevents effective weight transfer.<n>We identify Test-Time Training (TTT) as a linear-complexity architecture whose two-layer dynamic formulation is structurally aligned with Softmax attention.<n>We validate our approach by linearizing Stable Diffusion 3.5 and introduce SD3.5-T$5$ (Transformer To Test Time Training)
• Score: 54.52616866374482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While linear-complexity attention mechanisms offer a promising alternative to Softmax attention for overcoming the quadratic bottleneck, training such models from scratch remains prohibitively expensive. Inheriting weights from pretrained Transformers provides an appealing shortcut, yet the fundamental representational gap between Softmax and linear attention prevents effective weight transfer. In this work, we address this conversion challenge from two perspectives: architectural alignment and representational alignment. We identify Test-Time Training (TTT) as a linear-complexity architecture whose two-layer dynamic formulation is structurally aligned with Softmax attention, enabling direct inheritance of pretrained attention weights. To further align representational properties, including key shift-invariance and locality, we introduce key instance normalization and a lightweight locality enhancement module. We validate our approach by linearizing Stable Diffusion 3.5 and introduce SD3.5-T$^5$ (Transformer To Test Time Training). With only 1 hour of fine-tuning on 4$\times$H20 GPUs, SD3.5-T$^5$ achieves comparable text-to-image quality to the fine-tuned Softmax model, while accelerating inference by 1.32$\times$ and 1.47$\times$ at 1K and 2K resolutions.

Related papers

• Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware [5.066906328188419]
  Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost.<n>We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss.
  arXiv  Detail & Related papers  (2026-08-31T08:28:01Z)
• Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think [43.01990826833761]
  Vision-Language-Action (VLA) models pre-trained on massive video-robot datasets have revolutionized robotic manipulation.<n>Despite being trained on diverse physical trajectories, they exhibit severe layer-wise representational redundancy.<n>We introduce a structural compression pipeline that is entirely training-free.<n>Downstream fine-tuning of this streamlined architecture yields a 40-50% reduction in training time and up to 30% faster real-time inference.
  arXiv  Detail & Related papers  (2026-06-18T13:57:12Z)
• Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling [9.458812631271009]
  We introduce Nexusformer, which replaces linear $Q/K/V projections with a Nexus-Rank layer.<n>New capacity can be injected along two axes via zero-d blocks that preserve pretrained knowledge.<n>Experiments on language modeling and reasoning demonstrate that Nexusformer matches Tokenformer's perplexity benchmarks.
  arXiv  Detail & Related papers  (2026-04-21T06:54:16Z)
• Anisotropic Permeability Tensor Prediction from Porous Media Microstructure via Physics-Informed Progressive Transfer Learning with Hybrid CNN-Transformer [0.0]
  Prediction of permeability tensors from pore-scale microstructure images is essential for subsurface flow modeling, yet direct numerical simulation requires hours per sample.<n>A physics-informed deep learning framework is presented that resolves this bottleneck by combining a MaxViT hybrid CNN-Transformer architecture with progressive transfer learning and differentiable physical constraints.
  arXiv  Detail & Related papers  (2026-03-18T09:41:01Z)
• Scalable Generative Game Engine: Breaking the Resolution Wall via Hardware-Algorithm Co-Design [17.941176878609337]
  We bridge the gap between generative models and high-resolution neural simulations by introducing a scalable textitHardware-Algorithm Co-Design framework.<n>Our system delivers fluid 26.4 FPS and 48.3 FPS respectively, with an amortized effective latency of 2.7 ms.
  arXiv  Detail & Related papers  (2026-01-31T08:52:51Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10$\times$ [91.61519033897424]
  Native 4K video generation remains a critical challenge due to the quadratic computational explosion of full-attention as resolution increases.<n>This paper proposes a novel Transformer retrofit strategy termed $textbfT3-Video$ that significantly reduces compute requirements by optimizing their forward logic.<n>Results on 4K-VBench show that $textbfT3-Video$ substantially outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-12-15T16:25:39Z)
• Transformer Based Linear Attention with Optimized GPU Kernel Implementation [10.235738752130803]
  Linear attention (LA) mechanisms offer a linear time complexity of $O(ND2)$ and have demonstrated comparable accuracy to regular attention.<n>We propose a novel method for LA's forward and backward passes, along with a highly-optimized implementation.<n>We validate these improvements in both single-layer and end-to-end settings by training a 1.4 billion parameter language model.
  arXiv  Detail & Related papers  (2025-10-24T18:32:20Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up [64.38715211969516]
  We introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token.<n>Experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity.
  arXiv  Detail & Related papers  (2024-12-20T17:57:09Z)
• LinFusion: 1 GPU, 1 Minute, 16K Image [71.44735417472043]
  We introduce a low-rank approximation of a wide spectrum of popular linear token mixers. We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD. Experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion enables satisfactory and efficient zero-shot cross-resolution generation.
  arXiv  Detail & Related papers  (2024-09-03T17:54:39Z)
• Transformer Quality in Linear Time [95.2692237947444]
  We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality.
  arXiv  Detail & Related papers  (2022-02-21T18:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.