Dion: Distributed Orthonormalized Updates

• URL: http://arxiv.org/abs/2504.05295v3
• Date: Mon, 15 Sep 2025 16:02:53 GMT
• Title: Dion: Distributed Orthonormalized Updates
• Authors: Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford,
• Abstract summary: We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule.<n>It replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer.<n>The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings.
• Score: 27.66769374729482
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash with sharded weights in large-scale LLM training, causing high compute and communication cost. We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule that replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer, avoiding full-matrix reconstruction and integrating cleanly with weight sharding. The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings. On language models from 160M to 3B parameters, Dion retains the benefits of orthonormalized updates, while markedly reducing wall-clock time at scale, making it a practical optimizer for next-generation foundation models. Code is available at: https://github.com/microsoft/dion/

Related papers

• NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training [50.27276603708547]
  We show that despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines.<n>We propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure.
  arXiv  Detail & Related papers  (2026-03-04T00:10:14Z)
• TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers [24.534939825452884]
  TrasMuon preserves the near-isometric geometry of Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping.<n>We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers.<n> Empirical experiments on vision and language models demonstrate that TrasMuon converges faster than baselines.
  arXiv  Detail & Related papers  (2026-02-13T22:11:59Z)
• Stabilizing Native Low-Rank LLM Pretraining [24.2079184778031]
  Low-rank factorization offers a promising route to reduce training and inference costs.<n>We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights.<n>Our method enables stable, end-to-end factorized training with negligible overhead.
  arXiv  Detail & Related papers  (2026-02-12T21:33:14Z)
• Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL [16.40150726450328]
  In bandwidth-constrained decentralized environments, our approach achieves over 100x (14 GB to 108 MB) communication reduction.<n>We present a systematic empirical study of weight-update sparsity at both step-level and multi-step granularities.<n>We find that update sparsity is consistently high, frequently exceeding 99% across practically relevant settings.
  arXiv  Detail & Related papers  (2026-02-03T18:56:48Z)
• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation [58.3773038915023]
  Continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.<n>Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge.<n>We address this problem using a projected first-order method compatible with standard deep-dots used in vision-language models.
  arXiv  Detail & Related papers  (2026-01-31T13:27:02Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Dion2: A Simple Method to Shrink Matrix in Muon [19.766325230655173]
  We introduce Dion2, a much simpler method for shrinking the matrix involved in Muon's iteration compared to prior approaches.<n>At a high level, Dion2 selects a fraction of rows or columns at each and orthonormalizes only those.
  arXiv  Detail & Related papers  (2025-12-01T16:58:10Z)
• AuON: A Linear-time Alternative to Semi-Orthogonal Momentum Updates [0.0]
  We study the semi-orthogonal properties of momentum-based updates and develop a method to bound momentum updates under a spectral-norm trust region.<n>We propose AuON (Alternative Unit-norm momentum updates by Normalized nonlinear scaling), a linear-time that achieves strong performance without constructing semi-orthogonal matrices.<n>Our approach combines hyperbolic-cosine RMS scaling transformations with normalization, demonstrating both effectiveness and computational efficiency compared to Newton-Schulz methods.
  arXiv  Detail & Related papers  (2025-09-29T06:03:53Z)
• Low-rank Momentum Factorization for Memory Efficient Training [13.464518325870444]
  Momentum Factorized (MoFaSGD) maintains a dynamically updated low-rank SVD representation of the first-order momentum.<n>We demonstrate MoFaSGD's effectiveness on large language model benchmarks, achieving a competitive trade-off between memory reduction (e.g. LoRA) and performance.
  arXiv  Detail & Related papers  (2025-07-10T18:04:52Z)
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning [26.103555014247117]
  Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs)<n>We present AReaL, a fully asynchronous RL system that completely decouples generation from training.
  arXiv  Detail & Related papers  (2025-05-30T07:18:25Z)
• Advanced Relay-Based Collaborative Framework for Optimizing Synchronization in Split Federated Learning over Wireless Networks [4.235050593084289]
  Split Federated Learning (SFL) offers a promising approach for distributed model training in edge computing. We propose a collaborative SFL framework (CSFL) to optimize synchronization efficiency among users. We show that our proposed CSFL framework reduces synchronization delays and improves overall system throughput.
  arXiv  Detail & Related papers  (2025-03-18T22:11:54Z)
• Forget Forgetting: Continual Learning in a World of Abundant Memory [55.64184779530581]
  Continual learning has traditionally focused on minimizing exemplar memory.<n>This paper challenges this paradigm by investigating a more realistic regime.<n>We find that the core challenge shifts from stability to plasticity, as models become biased toward prior tasks and struggle to learn new ones.
  arXiv  Detail & Related papers  (2025-02-11T05:40:52Z)
• DeMo: Decoupled Momentum Optimization [6.169574689318864]
  Training large neural networks typically requires sharing between accelerators through specialized high-speed interconnects.<n>We introduce bfDecoupled textbfMomentum (DeMo), a fused magnitude and data parallel algorithm that reduces inter-accelerator communication requirements.<n> Empirical results show that models trained with DeMo match or exceed the performance of equivalent models trained with AdamW.
  arXiv  Detail & Related papers  (2024-11-29T17:31:47Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• Hyperdimensional Computing Empowered Federated Foundation Model over Wireless Networks for Metaverse [56.384390765357004]
  We propose an integrated federated split learning and hyperdimensional computing framework for emerging foundation models. This novel approach reduces communication costs, computation load, and privacy risks, making it suitable for resource-constrained edge devices in the Metaverse.
  arXiv  Detail & Related papers  (2024-08-26T17:03:14Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models [73.88009808326387]
  We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. We introduce Spectral Ortho Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity.
  arXiv  Detail & Related papers  (2024-05-31T17:43:35Z)
• Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging [1.4748100900619232]
  Federated Dynamic Averaging (FDA) is a communication-efficient DDL strategy. FDA reduces communication cost by orders of magnitude, compared to both traditional and cutting-edge algorithms.
  arXiv  Detail & Related papers  (2024-05-31T16:34:11Z)
• Diffusion Model Patching via Mixture-of-Prompts [17.04227271007777]
  Diffusion Model Patching (DMP) is a simple method to boost the performance of pre-trained diffusion models.<n>DMP inserts a small, learnable set of prompts into the model's input space while keeping the original model frozen.<n>DMP significantly enhances the FID of converged DiT-L/2 by 10.38% on FFHQ.
  arXiv  Detail & Related papers  (2024-05-28T04:47:54Z)
• AdaFisher: Adaptive Second Order Optimization via Fisher Information [22.851200800265914]
  First-order optimization methods are currently the mainstream in training deep neural networks (DNNs).s like Adam incorporate limited curvature information by employing the matrix preconditioning of the gradient during the training.<n>Despite their widespread, second-order optimization algorithms exhibit superior convergence properties compared to their first-order counterparts e.g. Adam and SGD.<n>We present emphAdaFisher--an adaptive second-order that leverages a emphdiagonal block-Kronecker approximation of the Fisher information matrix for adaptive gradient preconditioning.
  arXiv  Detail & Related papers  (2024-05-26T01:25:02Z)
• Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices [0.0]
  Ravnest facilitates decentralized training by efficiently organizing compute nodes into clusters. We have framed our asynchronous SGD loss function as a block structured optimization problem with delayed updates.
  arXiv  Detail & Related papers  (2024-01-03T13:07:07Z)
• Efficient Asynchronous Federated Learning with Sparsification and Quantization [55.6801207905772]
  Federated Learning (FL) is attracting more and more attention to collaboratively train a machine learning model without transferring raw data. FL generally exploits a parameter server and a large number of edge devices during the whole process of the model training. We propose TEASQ-Fed to exploit edge devices to asynchronously participate in the training process by actively applying for tasks.
  arXiv  Detail & Related papers  (2023-12-23T07:47:07Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks [44.37047471448793]
  In this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL) We propose an innovative PSL framework, namely, efficient parallel split learning (EPSL) to accelerate model training. We show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy.
  arXiv  Detail & Related papers  (2023-03-26T16:09:48Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Fast-Convergent Federated Learning via Cyclic Aggregation [10.658882342481542]
  Federated learning (FL) aims at optimizing a shared global model over multiple edge devices without transmitting (private) data to the central server. This paper utilizes cyclic learning rate at the server side to reduce the number of training iterations with increased performance. Numerical results validate that, simply plugging-in the proposed cyclic aggregation to the existing FL algorithms effectively reduces the number of training iterations with improved performance.
  arXiv  Detail & Related papers  (2022-10-29T07:20:59Z)
• Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization [13.89136187674851]
  In BO, a computationally cheap surrogate model is employed to learn the relationship between parameter configurations and their performance. We present an asynchronous-decentralized BO, wherein each worker runs a sequential BO and asynchronously communicates its results through shared storage. We scale our method without loss of computational efficiency with above 95% of worker's utilization to 1,920 parallel workers.
  arXiv  Detail & Related papers  (2022-07-01T15:07:56Z)
• Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism [56.78673028601739]
  We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT reduces communication by at least $100times$ and $20times$ during DP and MP, respectively. It improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.
  arXiv  Detail & Related papers  (2020-10-18T01:44:42Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.