LoCO: Low-rank Compositional Rotation Fine-tuning

• URL: http://arxiv.org/abs/2605.15916v1
• Date: Fri, 15 May 2026 12:54:56 GMT
• Title: LoCO: Low-rank Compositional Rotation Fine-tuning
• Abstract summary: Low-rank Compositional Orthogonal fine-tuning is a novel technique for adapting large-scale foundation models across natural language processing and computer vision.<n>We propose an approximation scheme that enables fully parallel computation of compositional rotations.<n>We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation.
• Score: 27.732021910046445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods.

Related papers

• New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework [52.69678300445233]
  Fine-tuning Large Language Models (LLMs) typically involves either full fine-tuning, which updates all model parameters, or.<n>Efficient Fine-Tuning (PEFT) adjusts a small subset of parameters.<n>We propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth-order and first-order optimization methods.
  arXiv  Detail & Related papers  (2026-04-10T22:39:38Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Orthogonal Model Merging [21.902153344111223]
  Merging finetuned Large Language Models (LLMs) has become increasingly important for integrating diverse capabilities into a single unified model.<n>We propose Orthogonal Model Merging (OrthoMerge) to preserve the geometric structure of the model's weights.
  arXiv  Detail & Related papers  (2026-02-05T17:57:14Z)
• From Coefficients to Directions: Rethinking Model Merging with Directional Alignment [66.99062575537555]
  We introduce a unified geometric framework, emphMerging with Directional Alignment (method), which aligns directional structures consistently in both the parameter and feature spaces.<n>Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-11-29T08:40:58Z)
• Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuning [12.174849960744211]
  Localized LoRA is a framework that models weight updates as a composition of low-rank approximations applied to structured blocks of the weight matrix.<n>We show that our method consistently achieves lower approximation error under matched parameter budgets.
  arXiv  Detail & Related papers  (2025-05-30T21:13:23Z)
• Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations [50.010924231754856]
  Adapting pre-trained foundation models for diverse downstream tasks is a core practice in artificial intelligence.<n>To overcome this, parameter-efficient fine-tuning (PEFT) methods like LoRA have emerged and are becoming a growing research focus.<n>We propose a generalization that extends matrix-based PEFT methods to higher-dimensional parameter spaces without compromising their structural properties.
  arXiv  Detail & Related papers  (2025-04-01T14:36:45Z)
• Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models [73.88009808326387]
  We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. We introduce Spectral Ortho Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity.
  arXiv  Detail & Related papers  (2024-05-31T17:43:35Z)
• Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation [32.371755315509574]
  Householder reflection adaptation (HRA) is a simple but effective adaptation method based on Householder reflections. HRA achieves superior performance with fewer learnable parameters when adapting large language models and conditional image generators.
  arXiv  Detail & Related papers  (2024-05-24T16:18:16Z)
• Convolutional Hough Matching Networks for Robust and Efficient Visual Correspondence [41.061667361696465]
  We introduce a Hough transform perspective on convolutional matching and propose an effective geometric matching algorithm, dubbed Convolutional Hough Matching (CHM) Our method sets a new state of the art on standard benchmarks for semantic visual correspondence, proving its strong robustness to challenging intra-class variations.
  arXiv  Detail & Related papers  (2021-09-11T08:39:41Z)
• Pushing the Envelope of Rotation Averaging for Visual SLAM [69.7375052440794]
  We propose a novel optimization backbone for visual SLAM systems. We leverage averaging to improve the accuracy, efficiency and robustness of conventional monocular SLAM systems. Our approach can exhibit up to 10x faster with comparable accuracy against the state-art on public benchmarks.
  arXiv  Detail & Related papers  (2020-11-02T18:02:26Z)
• Group Equivariant Deep Reinforcement Learning [4.997686360064921]
  We propose the use of Equivariant CNNs to train RL agents and study their inductive bias for transformation equivariant Q-value approximation. We demonstrate that equivariant architectures can dramatically enhance the performance and sample efficiency of RL agents in a highly symmetric environment.
  arXiv  Detail & Related papers  (2020-07-01T02:38:48Z)
• CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices [41.57234424773276]
  We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs. We further develop a novel Truncated CWY (or T-CWY) approach for Stiefel manifold parametrization. We apply our methods to train recurrent neural network architectures in the tasks of neural machine video prediction.
  arXiv  Detail & Related papers  (2020-04-18T17:58:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.