PVeRA: Probabilistic Vector-Based Random Matrix Adaptation

• URL: http://arxiv.org/abs/2512.07703v1
• Date: Mon, 08 Dec 2025 16:50:21 GMT
• Title: PVeRA: Probabilistic Vector-Based Random Matrix Adaptation
• Authors: Leo Fillioux, Enzo Ferrante, Paul-Henry Cournède, Maria Vakalopoulou, Stergios Christodoulidis,
• Abstract summary: We propose PVeRA, a probabilistic version of the VeRA adapter, which modifies the low-rank matrices of VeRA in a probabilistic manner.<n>A comprehensive evaluation was performed on the VTAB-1k benchmark and seven adapters, with PVeRA outperforming VeRA and other adapters.
• Score: 6.460933909139705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large foundation models have emerged in the last years and are pushing performance boundaries for a variety of tasks. Training or even finetuning such models demands vast datasets and computational resources, which are often scarce and costly. Adaptation methods provide a computationally efficient solution to address these limitations by allowing such models to be finetuned on small amounts of data and computing power. This is achieved by appending new trainable modules to frozen backbones with only a fraction of the trainable parameters and fitting only these modules on novel tasks. Recently, the VeRA adapter was shown to excel in parameter-efficient adaptations by utilizing a pair of frozen random low-rank matrices shared across all layers. In this paper, we propose PVeRA, a probabilistic version of the VeRA adapter, which modifies the low-rank matrices of VeRA in a probabilistic manner. This modification naturally allows handling inherent ambiguities in the input and allows for different sampling configurations during training and testing. A comprehensive evaluation was performed on the VTAB-1k benchmark and seven adapters, with PVeRA outperforming VeRA and other adapters. Our code for training models with PVeRA and benchmarking all adapters is available https://github.com/leofillioux/pvera.

Related papers

• PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint [3.5840378192062956]
  We introduce PHLoRA, a simple yet powerful method to extract low-rank adaptation adapters from full-rank fine-tuned models.<n>Unlike prior work that trains each adapter explicitly, our approach decouples fine-tuning from adapter generation.<n>Experiments on text, image, and video benchmarks using the Amazon Nova model family demonstrate that extracted adapters preserve high energy from the full weight delta, can be pruned safely, and yield negligible degradation in downstream task performance when re-merged.
  arXiv  Detail & Related papers  (2025-09-13T20:13:58Z)
• WeightLoRA: Keep Only Necessary Adapters [76.32368157312477]
  Low-rank adaptation ($texttLoRA$) adds trainable adapters to selected layers.<n>We propose a novel method, $textttWeightLoRA$, which overcomes this issue by adaptive selection of the most critical $textttLoRA$ heads.<n>We conduct experiments for a series of competitive benchmarks and DeBERTa, BART, and Llama models, comparing our method with different adaptive approaches.
  arXiv  Detail & Related papers  (2025-06-03T10:33:16Z)
• PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning [54.99373314906667]
  Self-supervised representation learning for point cloud has demonstrated effectiveness in improving pre-trained model performance across diverse tasks.<n>As pre-trained models grow in complexity, fully fine-tuning them for downstream applications demands substantial computational and storage resources.<n>We propose PointLoRA, a simple yet effective method that combines low-rank adaptation (LoRA) with multi-scale token selection to efficiently fine-tune point cloud models.
  arXiv  Detail & Related papers  (2025-04-22T16:41:21Z)
• DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection [32.369133126167085]
  Low-Rank Adaptation (LoRA) has gained popularity for its streamlined design by incorporating low-rank matrices into existing pre-trained models.<n>We propose DiffoRA, which enables adaptive adoption of the low-rank decomposition matrices.
  arXiv  Detail & Related papers  (2025-02-13T02:41:34Z)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation [53.88562288388169]
  A common strategy for. Efficient Fine-Tuning (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks. We propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product of a left unitary matrix, a diagonal matrix of scaling values, and a right unitary matrix.
  arXiv  Detail & Related papers  (2024-10-30T12:08:30Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• Adaptive Adapter Routing for Long-Tailed Class-Incremental Learning [55.384428765798496]
  New data exhibits a long-tailed distribution, such as e-commerce platform reviews. This necessitates continuous model learning imbalanced data without forgetting. We introduce AdaPtive Adapter RouTing (APART) as an exemplar-free solution for LTCIL.
  arXiv  Detail & Related papers  (2024-09-11T17:52:00Z)
• SARA: Singular-Value Based Adaptive Low-Rank Adaption [4.135688713311511]
  LoRA as a parameter-efficient fine-tuning(PEFT) method is widely used for not adding inference overhead. In this work, we first analyze the relationship between the performance of different layers and their ranks using SVD. Based on this, we design the Singular-Value Based Adaptive Low-Rank Adaption(SARA)
  arXiv  Detail & Related papers  (2024-08-06T16:39:42Z)
• Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
  Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. We propose a test-time Forward-Optimization Adaptation (FOA) method. FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
  arXiv  Detail & Related papers  (2024-04-02T05:34:33Z)
• ConvLoRA and AdaBN based Domain Adaptation via Self-Training [4.006331916849688]
  We propose Convolutional Low-Rank Adaptation (ConvLoRA) for multi-target domain adaptation. ConvLoRA freezes pre-trained model weights, adds trainable low-rank decomposition matrices to convolutional layers, and backpropagates the gradient. Our method has fewer trainable parameters and performs better or on-par with large independent fine-tuned networks.
  arXiv  Detail & Related papers  (2024-02-07T15:43:50Z)
• VeRA: Vector-based Random Matrix Adaptation [29.99581464596137]
  We present Vector-based Random Matrix Adaptation (VeRA), which reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.
  arXiv  Detail & Related papers  (2023-10-17T17:59:46Z)
• NOLA: Compressing LoRA using Linear Combination of Random Basis [22.76088132446952]
  We introduce NOLA, which overcomes the rank one lower bound present in LoRA. NOLA performs as well as LoRA models with much fewer number of parameters compared to LoRA with rank one, the best compression LoRA can archive.
  arXiv  Detail & Related papers  (2023-10-04T03:30:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.