Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

• URL: http://arxiv.org/abs/2609.15229v1
• Date: Mon, 14 Sep 2026 08:49:31 GMT
• Title: Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
• Abstract summary: We propose a pre-fine-tuning probing method for.<n>Efficient Fine-Tuning (PEFT) layer selection.<n>We observe a consistent correlation: layers with larger weight norms and higher condition numbers are usually more robust to perturbations.
• Score: 5.462799081500865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.

Related papers

• ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models [33.01358619125075]
  We propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate attention and activations in a fully frozen backbone.<n>Our method achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA.
  arXiv  Detail & Related papers  (2026-06-28T19:48:39Z)
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective [52.693471818837395]
  We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
  arXiv  Detail & Related papers  (2026-05-27T17:59:51Z)
• Visual prompting reimagined: The power of the Activation Prompts [72.85146015928626]
  We introduce a concept, termed activation prompt (AP), which extends the scope of the input-level VP.<n>AP is closely related to normalization tuning in convolutional neural networks and vision transformers.<n>We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers.
  arXiv  Detail & Related papers  (2026-04-07T20:28:24Z)
• Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Models [19.448467763421707]
  Large language models (LLMs) continue to grow, making parameter-efficient fine-tuning the default strategy for downstream adaptation.<n>Current practice typically applies PEFT uniformly across all layers, with limited understanding or leverage of layer selection.<n>This paper develops a unified projected residual view of PEFT on top of a frozen base model.
  arXiv  Detail & Related papers  (2026-02-03T21:05:55Z)
• TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating [8.102270371993411]
  We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices.<n>Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive.
  arXiv  Detail & Related papers  (2025-11-20T08:41:20Z)
• Beyond Outliers: A Study of Optimizers Under Quantization [82.75879062804955]
  We study impact of choice on model robustness under quantization.<n>We evaluate how model performance degrades when trained with different baselines.<n>We derive scaling laws for quantization-aware training under different parameters.
  arXiv  Detail & Related papers  (2025-09-27T21:15:22Z)
• SeWA: Selective Weight Average via Probabilistic Masking [51.015724517293236]
  We show that only a few points are needed to achieve better and faster convergence.<n>We transform the discrete selection problem into a continuous subset optimization framework.<n>We derive the SeWA's stability bounds, which are sharper than that under both convex image checkpoints.
  arXiv  Detail & Related papers  (2025-02-14T12:35:21Z)
• Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification [46.25272949924458]
  It is believed that fine-tuning the parameters of VLMs corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. We propose ClipFit, a method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that ClipFit can improve the performance of zero-shot CLIP by 7.27% average harmonic mean accuracy.
  arXiv  Detail & Related papers  (2024-09-25T08:07:18Z)
• WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models [8.481707805559589]
  We propose Weight-Generative Fine-Tuning (WeGeFT), a novel approach that learns to generate fine-tuning weights directly from the pretrained weights.<n>This design achieves multi-faceted efficiency in parameters, representations, compute, and memory, while maintaining or exceeding the performance of LoRA and its variants.
  arXiv  Detail & Related papers  (2023-12-01T16:33:57Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality [65.67315418971688]
  Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) are proposed. Experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-07-05T15:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.