HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

• URL: http://arxiv.org/abs/2603.16653v1
• Date: Tue, 17 Mar 2026 15:23:04 GMT
• Title: HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models
• Abstract summary: Heterogeneous Bottleneck Adapter (HeBA) is a unified architectural framework that introduces modality-specific inductive biases.<n>HeBA processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations.<n>Unlike standard expanding adapters, HeBA employs a compression bottleneck that explicitly forces the model to learn compact, robust features.
• Score: 3.8707695363745214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a "one-size-fits-all" architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D -> D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.

Related papers

• When Simplicity Wins: Bottleneck-Aware Context Modeling for Lightweight Semantic Segmentation [58.325533783810016]
  We propose SiConMo, a lightweight yet effective framework for semantic segmentation.<n>We show that simplicity arises from a key design principle: at very low computational budgets, the bottleneck is the most efficient stage to integrate local and global context.<n> experiments on ADE20K, PASCAL Context, Cityscapes, and COCO-Stuff demonstrate that SiConMo achieves a state-of-the-art accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2026-08-19T14:47:07Z)
• MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction [39.69838180175452]
  We propose MSVS-VAE, a hierarchical set-based VAE that closes this fidelity gap without sacrificing compactness.<n>Our key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling.<n>Extensive experiments on generative benchmarks, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAE consistently outperforms prior set-based and voxel-based VAEs, delivering approximately 10x faster decoding than prior set-based methods and approximately 10x higher compactness than voxel-based
  arXiv  Detail & Related papers  (2026-07-27T13:45:22Z)
• BA-T: An Iterative Transformer for Two-View Bundle Adjustment [53.16990349592508]
  We propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space.<n> experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations.<n>BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture.
  arXiv  Detail & Related papers  (2026-06-02T07:51:14Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation [28.481044299307346]
  Implicit representations lack explicit structural cues, whereas explicit ones preserve geometry but suffer from resolution limits and weak generalization.<n>We propose a novel pretraining framework that learns a hybrid representation-structural latent points.<n>The resulting compact latent preserves coarse structural tendencies, which do not encode precise geometry but capture richer rough shape and semantic information.
  arXiv  Detail & Related papers  (2026-05-20T14:48:01Z)
• Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners [34.02715256877424]
  unified models integrate multimodal understanding and generation within a single framework.<n>"understanding-generation gap" persists, where models can capture user intent but often fail to translate this semantic knowledge into precise pixel-level manipulation.<n>We propose a novel framework that empowers unified models to autonomously switch between generation strategies based on instruction complexity and model capability.
  arXiv  Detail & Related papers  (2026-05-14T11:27:46Z)
• ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation [3.8707695363745214]
  ReHARK is a training-free framework that reinterprets few-shot adaptation through global proximal regularization.<n>A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%.
  arXiv  Detail & Related papers  (2026-03-12T04:59:09Z)
• Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles [74.32932832937618]
  We introduce $textbfRigidSSL$ ($textitRigidity-Aware Self-Supervised Learning$), a geometric pretraining framework that front-loads geometry learning prior to finetuning.<n>Phase I (RigidSSL-Perturb) learns geometric priors from 432K structures from the AlphaFold Protein Structure Database with simulated perturbations.<n>Phase II (RigidSSL-MD) refines these representations on 1.3K molecular dynamics trajectories to capture physically realistic transitions.
  arXiv  Detail & Related papers  (2026-03-02T21:32:30Z)
• SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoning [30.87517633729756]
  SSR is a framework designed for Structured Scene Reasoning.<n>It seamlessly integrates 2D and 3D representations via a lightweight alignment mechanism.<n>It achieves state-of-the-art performance on multiple spatial intelligence benchmarks.
  arXiv  Detail & Related papers  (2026-02-28T02:05:35Z)
• CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval [54.15776146365823]
  Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text.<n>We propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components.
  arXiv  Detail & Related papers  (2026-01-07T09:21:38Z)
• Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models [33.99822400076112]
  Chunk-based sparse attention has emerged as a promising paradigm for extreme length generalization.<n>We present a systematic dissection of these models to identify the core components driving their performance.<n>We establish a new state-of-the-art for training-free length extrapolation, successfully generalizing models trained on a 4K context to 32 million tokens on RULER and BABILong.
  arXiv  Detail & Related papers  (2025-10-20T06:17:57Z)
• Light-SQ: Structure-aware Shape Abstraction with Superquadrics for Generated Meshes [60.92139345612904]
  We present Light-SQ, a novel superquadric-based optimization framework.<n>We propose a block-regrow-fill strategy guided by structure-aware volumetric decomposition.<n>Experiments demonstrate that Light-SQ enables efficient, high-fidelity, and editable shape abstraction with superquadrics.
  arXiv  Detail & Related papers  (2025-09-29T16:18:32Z)
• H3R: Hybrid Multi-view Correspondence for Generalizable 3D Reconstruction [39.22287224290769]
  H3R is a hybrid framework that integrates latent fusion with attention-based feature aggregation.<n>By integrating both paradigms, our approach enhances generalization while converging 2$times$ faster than existing methods.<n>Our method supports variable-number and high-resolution input views while demonstrating robust cross-dataset generalization.
  arXiv  Detail & Related papers  (2025-08-05T05:56:30Z)
• Cross-Layer Discrete Concept Discovery for Interpreting Language Models [13.842670153893977]
  Cross-layer VQ-VAE is a framework that uses vector quantization to map representations across layers.<n>Our approach uniquely combines top-k temperature-based sampling during quantization with EMA codebook updates.
  arXiv  Detail & Related papers  (2025-06-24T22:43:36Z)
• Cross-architecture universal feature coding via distribution alignment [88.73189953617594]
  We introduce a new research problem: cross-architecture universal feature coding (CAUFC)<n>We propose a two-step distribution alignment method. First, we design the format alignment method that CNN and Transformer features into a consistent 2D token format. Second, we propose the feature value alignment method that harmonizes statistical distributions via truncation and normalization.<n>As a first attempt to study CAUFC, we evaluate our method on the image classification task. Experimental results demonstrate that our method achieves superior rate-accuracy trade-offs compared to the architecture-specific baseline.
  arXiv  Detail & Related papers  (2025-06-15T06:14:02Z)
• ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions [91.55655961014027]
  3D semantic occupancy and flow prediction are fundamental to understanding scene scene.<n>This paper proposes a vision-based framework with three targeted improvements.<n>Our purely convolutional architecture establishes new SOTA performance on multiple benchmarks for both semantic occupancy and joint semantic-flow prediction.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.