Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

• URL: http://arxiv.org/abs/2603.08343v1
• Date: Mon, 09 Mar 2026 13:05:08 GMT
• Title: Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
• Abstract summary: We propose replacing the dense output projection in multi-head attention with a fixed, parameter-free Walsh Hadamard Transform.<n>Across different model sizes, we demonstrate that this structured substitution maintains comparable or slightly superior downstream task performance.<n>We observe that structured Hadamard-based models exhibit a steeper validation loss curve relative to training FLOPs, suggesting more favorable compute utilization during training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dense output projection in multi-head attention scales quadratically with model dimension, contributing significantly to parameter count, memory footprint, and inference cost. We propose replacing this projection with a fixed, parameter-free Walsh Hadamard Transform followed by a lightweight learnable affine rescaling, eliminating approximately 25 percent of attention parameters per block while preserving global cross head interaction through an orthogonal, norm-preserving transformation. Across different model sizes, we demonstrate that this structured substitution maintains comparable or slightly superior downstream task performance on standard benchmarks, while achieving up to 7 percent aggregate parameter reduction, 8.9 percent peak memory savings, and 6.6 percent throughput improvement at scale, with efficiency gains growing monotonically with model size, batch size, and sequence length. Interestingly, we observe that structured Hadamard-based models exhibit a steeper validation loss curve relative to training FLOPs compared to their dense counterparts, suggesting more favorable compute utilization during training.

Related papers

• RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers [8.085475675888045]
  RecurrentGPT is a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times.<n>Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate to modulate the recurrent update.<n>Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality.
  arXiv  Detail & Related papers  (2026-08-15T06:22:00Z)
• A general tensor-structured compression scheme for efficient large language models [8.85443988406591]
  We present a general tensor-structured compression scheme that replaces targeted dense linear layers with executable mixtures of tensor operators.<n>MixT is potentially applicable across Transformer-based large language models (LLMs) and other dense neural mappings.<n>At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1% and peak inference memory by 60.4%.
  arXiv  Detail & Related papers  (2026-05-25T02:00:41Z)
• JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation [46.64215658042213]
  We introduce Joint-Token (JTok) and Mixture of Joint-Token (JTok-M), which augment Transformer layers with modulation vectors retrieved from auxiliary embedding tables.<n>These vectors modulate the backbone via lightweight, element-wise operations, incurring negligible FLOPs overhead.<n>Our approach consistently reduces validation loss and significantly improves downstream task performance.
  arXiv  Detail & Related papers  (2026-01-31T16:15:18Z)
• EdgeFlex-Transformer: Transformer Inference for Edge Devices [2.1130318406254074]
  We propose a lightweight yet effective multi-stage optimization pipeline designed to compress and accelerate Vision Transformers (ViTs)<n>Our methodology combines activation profiling, memory-aware pruning, selective mixed-precision execution, and activation-aware quantization (AWQ) to reduce the model's memory footprint without requiring costly retraining or task-specific fine-tuning.<n>Experiments on CIFAR-10 demonstrate that the fully optimized model achieves a 76% reduction in peak memory usage and over 6x lower latency, while retaining or even improving accuracy compared to the original FP32 baseline.
  arXiv  Detail & Related papers  (2025-12-17T21:45:12Z)
• Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning [6.346469177254699]
  We propose a framework for structured weight sharing across transformer layers.<n>Inspired by dictionary learning in CNNs, we propose a framework for structured weight sharing across transformer layers.<n>Our approach decomposes attention projection matrices into shared dictionary atoms, reducing the attention module's parameters by 66.7%.
  arXiv  Detail & Related papers  (2025-08-06T16:06:43Z)
• Diversity-Guided MLP Reduction for Efficient Large Vision Transformers [62.33249256133204]
  Transformer models achieve excellent scaling property, where the performance is improved with the increment of model capacity.<n>Large-scale model parameters lead to an unaffordable cost of computing and memory.<n>We propose a Diversity-Guided Reduction (DGMR) method to significantly reduce the parameters of large vision transformers.
  arXiv  Detail & Related papers  (2025-06-10T08:59:27Z)
• Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference [0.0]
  We present Entropy Adaptive Decoding (EAD), a novel approach for efficient language model inference.<n>EAD switches between different-sized models based on prediction uncertainty.<n>We show remarkable efficiency gains across different model families.
  arXiv  Detail & Related papers  (2025-02-05T22:15:21Z)
• Tensor Product Attention Is All You Need [61.3442269053374]
  Product Attention (TPA) is a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly.<n>TPA achieves improved model quality alongside memory efficiency.<n>Based on TPA, we introduce the ProducT ATTion Transformer (T6), a new model architecture for sequence modeling.
  arXiv  Detail & Related papers  (2025-01-11T03:37:10Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation [53.675725490807615]
  We introduce SDPose, a new self-distillation method for improving the performance of small transformer-based models. SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs, while SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset.
  arXiv  Detail & Related papers  (2024-04-04T15:23:14Z)
• Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model [10.473819332984005]
  We propose a segmented recurrent transformer (SRformer) that combines segmented (local) attention with recurrent attention. The proposed model achieves $6-22%$ higher ROUGE1 scores than a segmented transformer and outperforms other recurrent transformer approaches.
  arXiv  Detail & Related papers  (2023-05-24T03:47:22Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• Highly Efficient Salient Object Detection with 100K Parameters [137.74898755102387]
  We propose a flexible convolutional module, namely generalized OctConv (gOctConv), to efficiently utilize both in-stage and cross-stages multi-scale features. We build an extremely light-weighted model, namely CSNet, which achieves comparable performance with about 0.2% (100k) of large models on popular object detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-12T07:00:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.