Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent

• URL: http://arxiv.org/abs/2609.08354v1
• Date: Tue, 08 Sep 2026 07:24:54 GMT
• Title: Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent
• Abstract summary: We build on the singular value decomposition factorization adapters to develop a framework based on Stein variational gradient descent (SVGD)<n>Our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.
• Score: 16.900185930373315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Several geometry-aware approaches to low-rank adaptation have emerged for parameter-efficient fine-tuning of large pre-trained models. These methods aim to take full advantage of the geometric structure of low-rank manifolds for improving the efficiency in subspace utilization and reducing redundancy by enforcing orthogonality constraints during optimization. The strong empirical results of these techniques have motivated further study into whether predictions from such geometry-based adaptation methods could be overconfident. In this paper, we build on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD). In this formulation, the low-rank matrices are transported along the Stiefel manifold to match the targeted distributions while retaining their crucial geometric structure. Since this geometry-aware SVGD approach provides multiple solutions during inference, it supports uncertainty quantification and produces better-calibrated adapters on the Stiefel manifold. Extensive experiments show that our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.

Related papers

• Convolutional Symmetric AutoEncoders: enhancing latent stability via differential geometry [0.0]
  This study builds upon works by arXiv:2307.15288v2 and arXiv:2506.11641v1 by extending representation consistency for convolutional layers.<n>We introduce a novel class of symmetric Convolutional AutoEncoders (CAEs) designed to embody the primary properties of manifold parametrization mappings.
  arXiv  Detail & Related papers  (2026-07-01T09:16:37Z)
• Geometry-Aware Neural Optimizer for Shape Optimization and Inversion [14.303592801147225]
  Geometry is central to PDE-governed systems, motivating shape optimization and inversion.<n>textbftextscGANO unifies geometry representation, field-level prediction, and automated optimization/inversion in a single latent-space loop.<n>textscGANO supports part-wise control through null-space projection and uses remeshing-free projection to accelerate geometry processing.
  arXiv  Detail & Related papers  (2026-05-06T03:51:22Z)
• Constrained Extreme Gradient Boosting for Adapting Reduced-Order Models [4.028552544912019]
  High-fidelity simulations, such as computational fluid dynamics and finite element analysis, are essential for modeling complex engineering systems.<n>Projection-based reduced-order models (ROMs) alleviate this cost by projecting the governing dynamics onto low-dimensional subspaces.<n>However, their performance can deteriorate under parameter variation, motivating the need for adaptive basis construction.
  arXiv  Detail & Related papers  (2026-05-05T16:58:09Z)
• Adaptive regularization parameter selection for high-dimensional inverse problems: A Bayesian approach with Tucker low-rank constraints [1.6936594801109557]
  Method reduces computational complexity by transforming variational inference from a high-dimensional space to a lower-dimensional core tensor space via Tucker decomposition.<n>Key innovation is the introduction of per-mode precision parameters, enabling adaptive regularization for anisotropic structures.<n>The approach scales to problems with 110,000 variables and outperforms existing methods by 0.73-2.09 dB in deblurring tasks and 6.75 dB in 3D heat conduction.
  arXiv  Detail & Related papers  (2026-03-17T02:23:44Z)
• Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization [8.201374511929538]
  This paper proposes a novel paradigm for machine learning that moves beyond traditional parameter optimization.<n>We optimize the metric tensor field on a manifold with a predefined topology, thereby dynamically shaping the geometric structure of the model space.<n>This work lays a solid foundation for constructing fully dynamic "meta-learners" capable of autonomously evolving their geometry and topology.
  arXiv  Detail & Related papers  (2025-10-30T01:53:32Z)
• On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization [57.179679246370114]
  A potential limitation of existing methods is the bias inherent in most perturbation estimators unless a stepsize is proposed.<n>We propose a novel family of unbiased gradient scaling estimators that eliminate bias while maintaining favorable construction.
  arXiv  Detail & Related papers  (2025-10-22T18:25:43Z)
• CP$^2$: Leveraging Geometry for Conformal Prediction via Canonicalization [51.716834831684004]
  We study the problem of conformal prediction (CP) under geometric data shifts.<n>We propose integrating geometric information--such as geometric pose--into the conformal procedure to reinstate its guarantees.
  arXiv  Detail & Related papers  (2025-06-19T10:12:02Z)
• Scaling Riemannian Diffusion Models [68.52820280448991]
  We show that our method enables us to scale to high dimensional tasks on nontrivial manifold. We model QCD densities on $SU(n)$ lattices and contrastively learned embeddings on high dimensional hyperspheres.
  arXiv  Detail & Related papers  (2023-10-30T21:27:53Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents [102.42623636238399]
  We identify fundamental geometric structures that underlie the problems of sampling, optimisation, inference and adaptive decision-making. We derive algorithms that exploit these geometric structures to solve these problems efficiently.
  arXiv  Detail & Related papers  (2022-03-20T16:23:17Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.