Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging

• URL: http://arxiv.org/abs/2512.13855v1
• Date: Mon, 15 Dec 2025 19:40:15 GMT
• Title: Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging
• Abstract summary: We introduce a novel PEFT framework that employs depth-aware scaling to increase adapter capacity from shallow to deep transformer layers.<n>Our method integrates lightweight bottleneck modules within CLIPSeg's vision and text encoders, with adapter dimensions dynamically scaled based on layer depth and semantic relevance.<n>Our approach establishes a new paradigm for efficient medical VLSM fine-tuning, enabling deployment in resource-constrained clinical environments.
• Score: 0.7865560760233441
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adapting Vision Language Segmentation Models (VLSMs) to medical imaging domains requires significant computational overhead when using conventional fine-tuning approaches. Existing Parameter-Efficient Fine-Tuning (PEFT) methods apply uniform adapter dimensions across all transformer layers, leading to suboptimal parameter allocation and reduced adaptation efficiency. We introduce Telescopic Adapters, a novel PEFT framework that employs depth-aware scaling to progressively increase adapter capacity from shallow to deep transformer layers. Our method integrates lightweight bottleneck modules within CLIPSeg's vision and text encoders, with adapter dimensions dynamically scaled based on layer depth and semantic relevance. Using only 613k trainable parameters--244x fewer than end-to-end fine-tuning, Telescopic Adapters achieve superior performance across five diverse medical datasets spanning polyp segmentation, skin lesion detection, and breast ultrasound imaging. Comprehensive ablation studies demonstrate that deeper layers require substantially more adaptation capacity than shallow layers, validating our telescopic scaling hypothesis. Our approach establishes a new paradigm for efficient medical VLSM fine-tuning, enabling deployment in resource-constrained clinical environments while maintaining competitive segmentation accuracy.

Related papers

• TAPE: A two-stage parameter-efficient adaptation framework for foundation models in OCT-OCTA analysis [7.475072200925591]
  We propose TAPE: A Two-stage Adaptation Framework via Efficient Fine-tuning.<n>This framework strategically decouples adaptation into domain alignment and task fitting for downstream segmentation.<n>It achieves state-of-the-art generalization performance across diverse pathologies.
  arXiv  Detail & Related papers  (2026-04-06T10:08:43Z)
• Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentation [6.91206648866302]
  We propose a depth-guided liver landmark segmentation framework integrating semantic and geometric cues via vision foundation encoders.<n>To efficiently adapt SAM2, we introduce SRFT-GaLore, a novel low-rank gradient projection method that replaces the computationally expensive SVD with a Subsampled Randomized Fourier Transform.<n>Our method achieves a 4.85% improvement in Dice Similarity Coefficient and a 11.78-point reduction in Average Symmetric Surface Distance compared to the D2GPLand.
  arXiv  Detail & Related papers  (2025-11-05T04:16:49Z)
• CoMViT: An Efficient Vision Backbone for Supervised Classification in Medical Imaging [0.3683202928838613]
  CoMViT is a compact and generalizable Vision Transformer architecture optimized for resource-constrained medical image analysis.<n>It achieves robust performance across twelve MedMNIST datasets while maintaining a lightweight design with only 4.5M parameters.
  arXiv  Detail & Related papers  (2025-10-31T12:49:13Z)
• F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning [28.598598268071587]
  We conduct the first systematic study of PEFT for pre-trained Large Operator Models (LOMs)<n>We observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning.<n>Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter)
  arXiv  Detail & Related papers  (2025-09-27T08:05:28Z)
• Accelerating 3D Photoacoustic Computed Tomography with End-to-End Physics-Aware Neural Operators [74.65171736966131]
  Photoacoustic computed tomography (PACT) combines optical contrast with ultrasonic resolution, achieving deep-tissue imaging beyond the optical diffusion limit.<n>Current implementations require dense transducer arrays and prolonged acquisition times, limiting clinical translation.<n>We introduce Pano, an end-to-end physics-aware model that directly learns the inverse acoustic mapping from sensor measurements to volumetric reconstructions.
  arXiv  Detail & Related papers  (2025-09-11T23:12:55Z)
• ClipGS: Clippable Gaussian Splatting for Interactive Cinematic Visualization of Volumetric Medical Data [51.095474325541794]
  We introduce ClipGS, an innovative Gaussian splatting framework with the clipping plane supported, for interactive cinematic visualization of medical data.<n>We validate our method on five volumetric medical data, and reach an average 36.635 PSNR rendering quality with 156 FPS and 16.1 MB model size.
  arXiv  Detail & Related papers  (2025-07-09T08:24:28Z)
• SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation [0.0]
  We propose a simple yet effective UNet-Transformer (seUNet-Trans) model for medical image segmentation. In our approach, the UNet model is designed as a feature extractor to generate multiple feature maps from the input images. By leveraging the UNet architecture and the self-attention mechanism, our model not only retains the preservation of both local and global context information but also is capable of capturing long-range dependencies between input elements.
  arXiv  Detail & Related papers  (2023-10-16T01:13:38Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)
• RetiFluidNet: A Self-Adaptive and Multi-Attention Deep Convolutional Network for Retinal OCT Fluid Segmentation [3.57686754209902]
  Quantification of retinal fluids is necessary for OCT-guided treatment management. New convolutional neural architecture named RetiFluidNet is proposed for multi-class retinal fluid segmentation. Model benefits from hierarchical representation learning of textural, contextual, and edge features.
  arXiv  Detail & Related papers  (2022-09-26T07:18:00Z)
• Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations [61.95114821573875]
  We introduce Plug-In Inversion, which relies on a simple set of augmentations and does not require excessive hyper- parameter tuning. We illustrate the practicality of our approach by inverting Vision Transformers (ViTs) and Multi-Layer Perceptrons (MLPs) trained on the ImageNet dataset.
  arXiv  Detail & Related papers  (2022-01-31T02:12:45Z)
• Medical Transformer: Gated Axial-Attention for Medical Image Segmentation [73.98974074534497]
  We study the feasibility of using Transformer-based network architectures for medical image segmentation tasks. We propose a Gated Axial-Attention model which extends the existing architectures by introducing an additional control mechanism in the self-attention module. To train the model effectively on medical images, we propose a Local-Global training strategy (LoGo) which further improves the performance.
  arXiv  Detail & Related papers  (2021-02-21T18:35:14Z)
• Efficient Semantic Image Synthesis via Class-Adaptive Normalization [116.63715955932174]
  Class-adaptive normalization (CLADE) is a lightweight but equally-effective variant that is only adaptive to semantic class. We introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE. The proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE.
  arXiv  Detail & Related papers  (2020-12-08T18:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.