AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning

• URL: http://arxiv.org/abs/2605.19767v1
• Date: Tue, 19 May 2026 12:39:30 GMT
• Title: AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
• Abstract summary: This paper shows that the bottleneck is a measurement-topology problem rather than a need for an external subspace.<n> AR1-ZO pairs alternating rank-$1$ atom queries with topology-aware scaling $=r$, restoring rank-invariant active signal.<n> Experiments on OPT and Qwen3 models validate the signal mechanism and show that AR1-ZO makes high-rank LoRA effective.
• Score: 17.98348342954707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zeroth-order (ZO) optimization enables large-language-model fine-tuning without storing backpropagation activations, while LoRA supplies compact trainable adapters. Combining them creates a rank paradox: increasing LoRA rank improves adapter capacity, but standard two-point ZO either perturbs a rank-dependent number of coordinates or, under atomwise updates, can make the finite-difference signal unobservable. This paper shows that the bottleneck is a measurement-topology problem rather than a need for an external subspace. LoRA already decomposes into matched rank-$1$ atoms, each a complete factor-coordinate block of dimension $d_\text{out}+d_\text{in}$. Querying one atom per step keeps the stored adapter rank $r$ while removing $r$ from the single-query perturbation dimension. The naive atomwise query is still miscalibrated: if it inherits canonical LoRA scaling $α/r$, the active finite-difference signal shrinks as $1/r$ and the active finite-difference signal-to-noise ratio (FD-SNR) as $1/r^2$, producing directional collapse under a fixed residual evaluation-noise floor. AR1-ZO pairs alternating rank-$1$ atom queries with topology-aware scaling $γ=αr$, restoring rank-invariant active signal without auxiliary bases, activation hooks, curvature estimates, or extra forward queries. Theory proves atom minimality, rank-independent active query dimension, directional collapse and restoration, and the remaining rank dependence as an amortized coverage cost. Experiments on OPT and Qwen3 models validate the signal mechanism and show that AR1-ZO makes high-rank LoRA effective among matched-budget ZO methods under the standard two-forward-pass query budget.

Related papers

• A Sharp Barrier for Consistent Submodular Maximization: Any Improvement over $2-\sqrt{2}$ Entails Exponential Queries or Linear Recourse [50.69285844345291]
  We study tradeoff between solution quality and stability when elements arrive over time.<n>Our algorithm has a future implementation for rational oracle$ with $O(varepsilon-1)$ recourse and separate the existence of universal-price certificates from their efficient algorithms.
  arXiv  Detail & Related papers  (2026-09-09T10:13:43Z)
• How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention [0.0]
  We provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention.<n>We also extend the analysis to fused multi-head LoRA and joint query/key updates.
  arXiv  Detail & Related papers  (2026-08-26T17:25:03Z)
• Adversarial Resilience of Poisson-Process Submodular Maximization over Matroids: From Robust Offline Optimization to Full-Bandit Learning [55.29259818039367]
  We study the nonnegative submodular subject to a general matroid when the offline algorithm is given an arbitrary controlled value oracle.<n>Our algorithm retains limiting approximation factors $1/e$ for non-monotone objectives and $1-1/e$ for monotone objectives.<n>As a consequence, the offline-to-online reduction yields single-bandit CMAB algorithms for general matroid-constrained submodular rewards.
  arXiv  Detail & Related papers  (2026-08-12T14:54:15Z)
• SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling [17.873016968298483]
  Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.<n>We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that re parameterizes a rank-rtot update as a sum of K static experts.
  arXiv  Detail & Related papers  (2026-06-27T03:25:36Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation [1.9996763865817948]
  Low-Rank Adaptation (LoRA) re-models a weight update as a product of two low-rank factors.<n>AdaPreLoRA fills this gap by adopting the Adaecker preconditioner $H_t$ on $W$ and selecting from the resulting factor-space solution family.
  arXiv  Detail & Related papers  (2026-05-09T06:37:44Z)
• Rethinking the Rank Threshold for LoRA Fine-Tuning [4.221888521641282]
  A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss.<n>We give three results that together reduce the prescribed rank to $r = 1$ for binary classification in this regime.
  arXiv  Detail & Related papers  (2026-05-05T13:09:46Z)
• Evolutionary Negative Module Pruning for Better LoRA Merging [46.397675691417675]
  Merging multiple Low-Rank Adaptation experts into a single backbone is a promising approach for efficient multi-task deployment.<n>Existing methods strive to alleviate interference via weight or subspace alignment.<n>We propose a plug-and-play LoRA pruning method to locate and exclude these modules prior to merging.
  arXiv  Detail & Related papers  (2026-04-20T03:13:18Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation [51.696027315573296]
  We propose a framework featuring a universal two-tiered reward mechanism designed to provide effective trajectory-level evaluation and dense step-level signals.<n>In traditional paradigms, rewards are determined solely by the final-turn feedback, which ignores the intermediate process and leads to ambiguous credit evaluation.<n>We show that our approach outperforms SOTA Arctic-Text2-R1-7B on BIRD 2.0 using identical models, propelling Text-to-reward convergence toward a robust multi-turn agent paradigm.
  arXiv  Detail & Related papers  (2026-03-17T06:26:36Z)
• Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation [0.0]
  Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models.<n>We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions.
  arXiv  Detail & Related papers  (2026-02-10T22:45:47Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models [10.827800772359844]
  We study the computational limits of Low-Rank Adaptation (LoRA) for finetuning transformer-based models using fine-grained complexity theory.<n>Our key observation is that the existence of low-rank decompositions within the gradient computation of LoRA adaptation leads to possible algorithmic speedup.
  arXiv  Detail & Related papers  (2024-06-05T10:44:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.