Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

• URL: http://arxiv.org/abs/2605.12492v1
• Date: Tue, 12 May 2026 17:59:34 GMT
• Title: Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
• Abstract summary: We introduce Pion, a spectrum-preserving for large language model (LLM) training based on orthogonal equivalence transformation.<n>We derive the Pion update rule systematically examine its design choices, and analyze its convergence behavior along with several key properties.<n> Empirical results show that Pion offers a stable and competitive alternative to the standard LLMs for both pretraining and finetuning.
• Score: 30.86228838205224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Related papers

• Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers [3.433766572511366]
  A striking geometric disparity has long persisted in the practice of deep learning.<n>We introduce a symmetry-compatible principle for design: the gradient update rule should be equivariant under the parameter group acting on the corresponding weight block.
  arXiv  Detail & Related papers  (2026-05-18T09:17:26Z)
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization [6.787340999768117]
  Large language models (LLMs) are convex convex by minimizing crossentropy loss for next-shift prediction.<n>We show that when the target tokens exhibit a cyclic symmetry, the optimal logit matrix is exactly circulant and the Gramtoken both form circulant output as well.
  arXiv  Detail & Related papers  (2026-05-12T21:10:34Z)
• Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning [15.604900995606451]
  We propose Muon-OGD, a spectral-norm-aware continual learning framework.<n>We evaluate the proposed method on standard continual learning benchmarks.
  arXiv  Detail & Related papers  (2026-05-09T13:42:08Z)
• Linear Reservoir: A Diagonalization-Based Optimization [1.7258249784556916]
  diagonalization-based optimization for Linear Echo State Networks (ESNs)<n>We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from O(N2) to O(N)
  arXiv  Detail & Related papers  (2026-02-23T12:58:34Z)
• Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning [10.647088281181222]
  SpecMuon is a spectral-aware, multi-mode gradient flow for physics-informed learning.<n>It regulates step sizes according to the global loss energy while preserving Muon's scale-balancing properties.<n>It achieves faster convergence and improved stability compared with Adam AdamW.
  arXiv  Detail & Related papers  (2026-02-18T03:56:20Z)
• Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation [56.361076943802594]
  CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG dataset, and the advantage remains large in few-step generation.
  arXiv  Detail & Related papers  (2026-02-16T18:58:55Z)
• Variational Entropic Optimal Transport [67.76725267984578]
  We propose Variational Entropic Optimal Transport (VarEOT) for domain translation problems.<n>VarEOT is based on an exact variational reformulation of the log-partition $log mathbbE[exp(cdot)$ as a tractable generalization over an auxiliary positive normalizer.<n> Experiments on synthetic data and unpaired image-to-image translation demonstrate competitive or improved translation quality.
  arXiv  Detail & Related papers  (2026-02-02T15:48:44Z)
• FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer [30.184978506988767]
  We introduce FISMO, which incorporates anisotropic neuralotropic geometry information through Fisher information geometry.<n> FISMO achieves superior efficiency and final performance compared to established baselines.
  arXiv  Detail & Related papers  (2026-01-29T14:05:04Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation [53.88562288388169]
  A common strategy for. Efficient Fine-Tuning (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks. We propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product of a left unitary matrix, a diagonal matrix of scaling values, and a right unitary matrix.
  arXiv  Detail & Related papers  (2024-10-30T12:08:30Z)
• Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models [73.88009808326387]
  We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. We introduce Spectral Ortho Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity.
  arXiv  Detail & Related papers  (2024-05-31T17:43:35Z)
• Multiplicative Updates for Online Convex Optimization over Symmetric Cones [28.815822236291392]
  We introduce the Symmetric-Cone Multiplicative Weights Update (SCMWU), a projection-free algorithm for online optimization over the trace-one slice of an arbitrary symmetric cone. We show that SCMWU is equivalent to Follow-the-Regularized-Leader and Online Mirror Descent with symmetric-cone negative entropy as regularizer.
  arXiv  Detail & Related papers  (2023-07-06T17:06:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.