Training Transformers in Cosine Coefficient Space

• URL: http://arxiv.org/abs/2604.04440v2
• Date: Thu, 09 Apr 2026 03:32:05 GMT
• Title: Training Transformers in Cosine Coefficient Space
• Abstract summary: A transformer trained from scratch on character-level Shakespeare reaches validation loss $1.604$ at $K = mn/2$.<n>A rank-48 LoRA factorization at the same trainable parameter count reaches only $1.801$.<n>A random orthonormal basis matches the DCT within noise at $K = mn/2$, and a compression sweep through $K = mn/10$ and $K = mn/20$ shows that subspaces that can host high-rank matrices keep the loss low.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear layers hold most of a transformer's parameters. We replace each linear layer with one that stores $K$ out of $mn$ two-dimensional DCT coefficients per weight matrix and reconstructs the full matrix through an inverse DCT at every forward pass; the $K$ coefficients are the trainable parameters. A 4-layer, 128-dim transformer trained from scratch on character-level Shakespeare reaches validation loss $1.604$ at $K = mn/2$, against $1.580$ for a standard dense baseline -- a gap of $+0.024$ at half the trainable parameter count, within the terminal-epoch variation of the dense run. A rank-48 LoRA factorization at the same trainable parameter count reaches only $1.801$ ($+0.221$). The structural advantage of sparse-coefficient over low-rank parameterizations at matched $K$ is qualitative. We identify rank flexibility as the mechanism. A random orthonormal basis matches the DCT within noise at $K = mn/2$, and a compression sweep through $K = mn/10$ and $K = mn/20$ shows that subspaces that can host high-rank matrices keep the loss low, while subspaces that flatten into a low-rank block (zigzag-selection variants) converge onto the observed stable rank \emph{and} the loss line of the rank-48 LoRA reference in lock-step. Among these orthonormal bases, the DCT is preferred because its separable fast transform admits a fused reconstruction kernel: the materialized weight matrix never leaves on-chip memory, so the parameter saving translates into a bandwidth saving as well.

Related papers

• Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m [0.0]
  Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation.<n>We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates.<n>The phenomenon is invisible to the standard SAE metric.
  arXiv  Detail & Related papers  (2026-05-23T13:37:59Z)
• Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics [0.0]
  Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse.<n>We show that weight decay acts as a scalar empirical control parameter for these regimes.<n>We introduce online diagnostics, mean pairwise attentionhead cosine similarity and entropy standard deviation, that track training dynamics.
  arXiv  Detail & Related papers  (2026-05-19T19:48:40Z)
• Architecture Shape Governs QNN Trainability: Jacobian Null Space Growth and Parameter Efficiency [3.1094222618006664]
  We show that trainability varies substantially with architecture shape $(N,L)$ at fixed $E$.<n>We identify structural rank deficiency of the coefficient matching Jacobian $J$ as the mechanism responsible.
  arXiv  Detail & Related papers  (2026-05-07T09:52:45Z)
• Scaling Federated Linear Contextual Bandits via Sketching [49.12000877146222]
  This paper proposes Federated Sketch Contextual Linear Bandits (FSCLB)<n>Experiments on both synthetic and real-world datasets show that FSCLB significantly reduces computational and communication costs by over 90 %.
  arXiv  Detail & Related papers  (2026-05-01T08:22:06Z)
• The Spectral Edge Thesis: A Mathematical Framework for Intra-Signal Phase Transitions in Neural Network Training [0.0]
  We show that phase transitions in neural network training are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates.<n>Adiabatic parameter $mathcalA = |G|_F / (, g2)$ controls circuit stability: $mathcalA ll 1$ (plateau), $mathcalA sim 1$ (phase transition), $mathcalA gg 1$ (forgetting)
  arXiv  Detail & Related papers  (2026-03-30T20:10:22Z)
• Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT [0.0]
  Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive.<n>We study Matrix Product Operator decomposition as a principled compression method for transformers.<n>MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi.
  arXiv  Detail & Related papers  (2026-03-30T14:57:47Z)
• Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers [0.0]
  We propose replacing the dense output projection in multi-head attention with a fixed, parameter-free Walsh Hadamard Transform.<n>Across different model sizes, we demonstrate that this structured substitution maintains comparable or slightly superior downstream task performance.<n>We observe that structured Hadamard-based models exhibit a steeper validation loss curve relative to training FLOPs, suggesting more favorable compute utilization during training.
  arXiv  Detail & Related papers  (2026-03-09T13:05:08Z)
• Structured Multidimensional Representation Learning for Large Language Models [0.0]
  Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks.<n>We introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors.<n>We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduced embeddings.
  arXiv  Detail & Related papers  (2026-03-05T22:34:45Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• Evolution Strategies at the Hyperscale [57.75314521465674]
  We introduce EGGROLL, an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes.<n>ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives.<n>EGGROLL overcomes these bottlenecks by generating random matrices $Ain mathbbRmtimes r, Bin mathbbRntimes r$ with $rll min(m,n)$ to form a low-rank matrix perturbation $A Btop$
  arXiv  Detail & Related papers  (2025-11-20T18:56:05Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Diversity-Guided MLP Reduction for Efficient Large Vision Transformers [62.33249256133204]
  Transformer models achieve excellent scaling property, where the performance is improved with the increment of model capacity.<n>Large-scale model parameters lead to an unaffordable cost of computing and memory.<n>We propose a Diversity-Guided Reduction (DGMR) method to significantly reduce the parameters of large vision transformers.
  arXiv  Detail & Related papers  (2025-06-10T08:59:27Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Dynamic Layer Tying for Parameter-Efficient Transformers [65.268245109828]
  We employ Reinforcement Learning to select layers during training and tie them together. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method.
  arXiv  Detail & Related papers  (2024-01-23T14:53:20Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• $\rm SP^3$: Enhancing Structured Pruning via PCA Projection [24.74632458058136]
  This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d. Experiments show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio.
  arXiv  Detail & Related papers  (2023-08-31T05:40:14Z)
• Hybrid Model-based / Data-driven Graph Transform for Image Coding [54.31406300524195]
  We present a hybrid model-based / data-driven approach to encode an intra-prediction residual block. The first $K$ eigenvectors of a transform matrix are derived from a statistical model, e.g., the asymmetric discrete sine transform (ADST) for stability. Using WebP as a baseline image, experimental results show that our hybrid graph transform achieved better energy compaction than default discrete cosine transform (DCT) and better stability than KLT.
  arXiv  Detail & Related papers  (2022-03-02T15:36:44Z)
• End-to-End Multi-speaker Speech Recognition with Transformer [88.22355110349933]
  We replace the RNN-based encoder-decoder in the speech recognition model with a Transformer architecture. We also modify the self-attention component to be restricted to a segment rather than the whole sequence in order to reduce computation.
  arXiv  Detail & Related papers  (2020-02-10T16:29:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.