Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

• URL: http://arxiv.org/abs/2607.21973v1
• Date: Fri, 24 Jul 2026 04:39:55 GMT
• Title: Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation
• Abstract summary: Prompted Information Bottlenecks (PIB) is a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path.<n>PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k.<n>These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models.
• Score: 42.92624141712034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision foundation models are increasingly reused as frozen backbones for downstream visual recognition, making parameter-efficient adaptation a central problem. Prompt-based adaptation, including Visual Prompt Tuning (VPT), provides a lightweight way to specialize these models, but its layer-wise behavior remains poorly understood: performance is sensitive to prompt depth, placement, and task distribution, and gains on standard in-domain benchmarks do not always translate into robust generalization. We argue that this limitation is not solely an optimization issue, but a layer-wise information allocation issue: existing prompt-based methods lack principled control over what prompt-conditioned representations should preserve, suppress, and propagate across depth. Inspired by the Information Bottleneck principle, we introduce Prompted Information Bottlenecks (PIB), a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path. The key idea is that effective adaptation should be minimal yet sufficient, retaining task-relevant local evidence in earlier layers while progressively discarding nuisance factors and redundant details in deeper layers. Extensive experiments show that PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k, while tuning only 0.35% parameters on average across the main settings. Beyond benchmark accuracy, PIB helps explain the non-monotonic behavior of prompt capacity scaling, reduces shortcut reliance, and improves robustness under distribution shift and fine-grained recognition settings. These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models. Our code is available at https://github.com/itsnotacie/MM-26-PIB

Related papers

• Scaling Adaptive Depth with Norm-Agnostic Residual Networks [9.219329760729629]
  NAG is a norm-agnostic residual architecture that separates magnitude from directional information in the residual stream.<n>NAG introduces only a negligible number of additional parameters and relies on simple operations that are easily kernel-fusible.<n>We show that this architecture outperforms baseline Transformers, with gains that increase substantially as depth grows.
  arXiv  Detail & Related papers  (2026-06-15T02:05:37Z)
• Accelerating Vision Foundation Models with Drop-in Depthwise Convolution [51.50107862675191]
  We introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for attention heads.<n>Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation.
  arXiv  Detail & Related papers  (2026-05-21T08:07:23Z)
• Channel-Level Relation to Attentive Aggregation with Neighborhood-Homogeneity Constraint for Point Cloud Analysis [14.63838637349137]
  In 3D point cloud understanding, the core challenge lies in accurately capturing discriminative features within complex neighborhoods.<n>Existing methods explore feature correlation discrimination but are limited to point-level spatial distribution or channel responses.<n>We propose PointCRA, a novel network with a channel-level metric-based enhancement mechanism.
  arXiv  Detail & Related papers  (2026-05-04T08:59:36Z)
• Monocular Depth Estimation From the Perspective of Feature Restoration: A Diffusion Enhanced Depth Restoration Approach [17.258737579616856]
  Monocular Depth Estimation (MDE) is a fundamental computer vision task with important applications in 3D vision.<n>In this paper, we first investigate the above problem and show that there is still substantial potential in the current framework if encoder features can be improved.<n> Experiments demonstrate that the proposed method achieves better performance than the state-of-the-art methods on various datasets.
  arXiv  Detail & Related papers  (2026-04-09T00:13:53Z)
• Visual prompting reimagined: The power of the Activation Prompts [72.85146015928626]
  We introduce a concept, termed activation prompt (AP), which extends the scope of the input-level VP.<n>AP is closely related to normalization tuning in convolutional neural networks and vision transformers.<n>We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers.
  arXiv  Detail & Related papers  (2026-04-07T20:28:24Z)
• Beyond Weight Adaptation: Feature-Space Domain Injection for Cross-Modal Ship Re-Identification [3.6907522136316975]
  Cross-Modality Ship Re-Identification (CMS Re-ID) is critical for achieving all-day and all-weather maritime target tracking.<n>We explore the potential of Vision Foundation Models (VFMs) in bridging modality gaps.<n>We propose a novel PEFT strategy termed Domain Representation Injection (DRI)
  arXiv  Detail & Related papers  (2025-12-24T02:30:23Z)
• Glocal Information Bottleneck for Time Series Imputation [70.41814118117311]
  Time Series Imputation aims to recover missing values in temporal data.<n>Existing models typically optimize the point-wise reconstruction loss, focusing on recovering numerical values (local information)<n>We propose a new training paradigm, Glocal Information Bottleneck (Glocal-IB)
  arXiv  Detail & Related papers  (2025-10-06T15:24:44Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• IB-AdCSCNet:Adaptive Convolutional Sparse Coding Network Driven by Information Bottleneck [4.523653503622693]
  We introduce IB-AdCSCNet, a deep learning model grounded in information bottleneck theory. IB-AdCSCNet seamlessly integrates the information bottleneck trade-off strategy into deep networks. Experimental results on CIFAR-10 and CIFAR-100 datasets demonstrate that IB-AdCSCNet not only matches the performance of deep residual convolutional networks but also outperforms them when handling corrupted data.
  arXiv  Detail & Related papers  (2024-05-23T05:35:57Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.