Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

• URL: http://arxiv.org/abs/2606.27748v2
• Date: Mon, 29 Jun 2026 02:38:34 GMT
• Title: Flexformer: Flexible Linear Transformer with Learnable Attention Kernel
• Abstract summary: We propose Flexformer, a flexible linear Transformer that learns attention kernels in a fully data-driven manner.<n>Experiments on language modeling and sequence classification demonstrate that Flexformer consistently outperforms baselines.
• Score: 8.917992463399969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences. Kernel-based linear attention reduces this complexity but typically relies on fixed or weakly learnable kernels, restricting expressiveness and performance. In this work, we propose Flexformer, a flexible linear Transformer that learns attention kernels in a fully data-driven manner. Flexformer builds on random Fourier feature-based linear attention and treats spectral frequencies as trainable parameters, enabling the model to learn a broad family of attention kernels. We develop both stationary and nonstationary variants, with the latter offering strictly greater expressiveness. Extensive experiments on language modeling and sequence classification demonstrate that Flexformer consistently outperforms baselines. Moreover, Flexformer can be effectively distilled from pretrained Transformers to recover softmax attention and exhibits strong kernel transferability across domains, achieving both high efficiency and competitive performance on long-sequence tasks.

Related papers

• Flex-Forcing: Towards a Unified Autoregressive and Bidirectional Video Diffusion Model [88.42070387375679]
  We introduce Flex-Forcing, a unified training and inference framework that enables a video diffusion model to seamlessly operate under both bidirectional and autoregressive generation regimes.<n>Experiments on multiple video generation benchmarks demonstrate that Flex-Forcing achieves consistently better video quality, long-video stability than strong baselines with a rigid inference schedule.
  arXiv  Detail & Related papers  (2026-07-03T17:34:14Z)
• Elastic Spiking Transformers for Efficient Gesture Understanding [6.52225846892392]
  Spiking Neural Networks (SNNs) offer energy-efficient processing of event-based sensor data for healthcare applications.<n>We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm.<n>We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset.
  arXiv  Detail & Related papers  (2026-05-04T12:35:52Z)
• Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers [69.60693631859411]
  This paper explores controllable diffusion models built upon linear attention architectures.<n>We propose a novel controllable diffusion framework tailored for linear attention backbones like SANA.<n>Our approach achieves state-of-the-art controllable generation performance based on linear-attention models.
  arXiv  Detail & Related papers  (2026-03-29T12:31:34Z)
• FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation [33.208889745659825]
  Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs.<n>We propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that supports rank pruning and expansion under a global budget.<n>Experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks.
  arXiv  Detail & Related papers  (2026-01-30T12:25:47Z)
• Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation [25.321469825055733]
  Diffusion transformers (DiTs) have emerged as a powerful architecture for high-fidelity image generation.<n>DyDiLA is a novel linear attention formulation that enhances the effectiveness of LiTs by mitigating the oversmoothing issue and improving generation quality.<n>To capitalize on DyDiLA, we introduce a refined LiT, termed DyDi-LiT, that systematically incorporates our advancements.
  arXiv  Detail & Related papers  (2026-01-20T07:33:16Z)
• Lizard: An Efficient Linearization Framework for Large Language Models [113.87302474262798]
  We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures.<n>Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality.<n>We show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark.
  arXiv  Detail & Related papers  (2025-07-11T21:19:18Z)
• Flextron: Many-in-One Flexible Large Language Model [85.93260172698398]
  We introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. We present a sample-efficient training method and associated routing algorithms for transforming an existing trained LLM into a Flextron model. We demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining.
  arXiv  Detail & Related papers  (2024-06-11T01:16:10Z)
• FLatten Transformer: Vision Transformer using Focused Linear Attention [80.61335173752146]
  Linear attention offers a much more efficient alternative with its linear complexity. Current linear attention approaches either suffer from significant performance degradation or introduce additional computation overhead. We propose a novel Focused Linear Attention module to achieve both high efficiency and expressiveness.
  arXiv  Detail & Related papers  (2023-08-01T10:37:12Z)
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers [55.90468016961356]
  We propose an efficient token mixer that learns to mix in the Fourier domain. AFNO is based on a principled foundation of operator learning. It can handle a sequence size of 65k and outperforms other efficient self-attention mechanisms.
  arXiv  Detail & Related papers  (2021-11-24T05:44:31Z)
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding [63.539333383965726]
  We propose a novel way to accelerate attention calculation for Transformers with relative positional encoding (RPE) Based upon the observation that relative positional encoding forms a Toeplitz matrix, we mathematically show that kernelized attention with RPE can be calculated efficiently using Fast Fourier Transform (FFT)
  arXiv  Detail & Related papers  (2021-06-23T17:51:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.