T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis

• URL: http://arxiv.org/abs/2510.27265v1
• Date: Fri, 31 Oct 2025 08:05:40 GMT
• Title: T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis
• Abstract summary: Existing model-merging techniques fail to deliver consistent gains across diverse medical modalities.<n>We introduce Test-Time Task adaptive merging (T3), a backpropagation-free framework that computes per-sample coefficients.<n>We present a rigorous cross-evaluation protocol spanning in-domain, base-to-novel, and corruptions across four modalities.
• Score: 15.624549727053475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In medical imaging, vision-language models face a critical duality: pretrained networks offer broad robustness but lack subtle, modality-specific characteristics, while fine-tuned expert models achieve high in-distribution accuracy yet falter under modality shift. Existing model-merging techniques, designed for natural-image benchmarks, are simple and efficient but fail to deliver consistent gains across diverse medical modalities; their static interpolation limits reliability in varied clinical tasks. To address this, we introduce Test-Time Task adaptive merging (T^3), a backpropagation-free framework that computes per-sample interpolation coefficients via the Jensen-Shannon divergence between the two models' output distributions. T^3 dynamically preserves local precision when models agree and defers to generalist robustness under drift. To overcome the inference costs of sample-wise merging, we further propose a batch-wise extension, T^3_B, that computes a merging coefficient across a batch of samples, dramatically reducing computational bottleneck. Recognizing the lack of a standardized medical-merging benchmark, we present a rigorous cross-evaluation protocol spanning in-domain, base-to-novel, and corruptions across four modalities. Empirically, T^3 sets new state-of-the-art in Top-1 accuracy and error reduction, outperforming strong baselines while maintaining efficiency, paving the way for adaptive MVLM deployment in clinical settings. Our code is available at https://github.com/Razaimam45/TCube.

Related papers

• D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities [54.87947751720332]
  We propose a novel segmentation model D3Seg to maintain stable performance under missing-modality settings.<n>The proposed model achieves approximately 1.5-2.0% Dice improvement on enhancing tumor (ET) and around 1.0% on tumor core (TC) across multiple missing modality configurations.
  arXiv  Detail & Related papers  (2026-05-21T09:55:42Z)
• Towards Generalized Image Manipulation Localization via Score-based Model [38.03898206912958]
  We propose DiffIML, a novel framework that introduces score-based generative modeling to IML.<n>Under this formulation, diffusion models serve as an effective numerical solver for the learned score function.<n>Experiments on eight non-generative and three generative benchmarks demonstrate that DiffIML consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-16T08:39:54Z)
• Align then Refine: Text-Guided 3D Prostate Lesion Segmentation [5.284052813431228]
  Volumetric methods must combine multiple modalities while ensuring anatomical consistency.<n>Current models struggle to integrate cross-modal information reliably.<n>We propose a new multi-encoder U-Net architecture incorporating three key innovations.
  arXiv  Detail & Related papers  (2026-04-20T18:13:52Z)
• The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging [3.597779662054083]
  Model merging under unseen test-time distribution shifts often renders naive strategies, such as mean averaging unreliable.<n>We introduce an entropy-adaptive, fully online model-merging method that yields a batch-specific merged model via only forward passes.<n>We extensively evaluate our method with state-of-the-art baselines using two backbones across nine medical and natural-domain generalization image classification datasets.
  arXiv  Detail & Related papers  (2026-02-24T21:06:19Z)
• Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance [51.532841645285835]
  We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution.<n>We show it can fail to faithfully unlearn with finite samples when the forget set represents a sharp, concentrated data distribution.<n>We introduce Temper-Then-Tilt Unlearning (T3-Unlearning), which freezes the base model and applies a two-step inference procedure.
  arXiv  Detail & Related papers  (2026-02-10T19:08:40Z)
• Tail-Aware Post-Training Quantization for 3D Geometry Models [58.79500829118265]
  Post-Training Quantization (PTQ) enables efficient inference without retraining.<n>PTQ fails to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead.<n>We propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline for 3D geometric learning.
  arXiv  Detail & Related papers  (2026-02-02T07:21:15Z)
• Convolutional Surrogate for 3D Discrete Fracture-Matrix Tensor Upscaling [0.0]
  We develop a surrogate model that predicts the equivalent hydraulic conductivity tensor Keq from a voxelized 3D domain.<n>Three surrogates are trained on data generated by DFM simulations, each corresponding to a different fracture-to-matrix conductivity contrast.<n>The trained models achieve high accuracy, with normalized root-mean-square errors below 0.22 across most test cases.
  arXiv  Detail & Related papers  (2026-01-19T23:53:58Z)
• Robust Variational Model Based Tailored UNet: Leveraging Edge Detector and Mean Curvature for Improved Image Segmentation [7.638424494500011]
  This paper presents a robust version of Variational Model Based UNet (VM_TUNet)<n>VM_TUNet is a hybrid framework that integrates variational methods with deep learning.<n> experiments on three benchmark datasets indicate that the proposed method achieves a balanced trade-off between performance and computational efficiency.
  arXiv  Detail & Related papers  (2025-12-08T14:33:52Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• Scalable, Explainable and Provably Robust Anomaly Detection with One-Step Flow Matching [14.503330877000758]
  Time-Conditioned Contraction Matching is a novel method for semi-supervised anomaly detection in tabular data.<n>It is inspired by flow matching, a recent generative modeling framework that learns velocity fields between probability distributions.<n>Extensive experiments on the ADBench benchmark show that TCCM strikes a favorable balance between detection accuracy and inference cost.
  arXiv  Detail & Related papers  (2025-10-21T06:26:38Z)
• Score Distillation of Flow Matching Models [67.86066177182046]
  We extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models.<n>SiD works out of the box across these models, in both data-free and data-aided settings.<n>This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models.
  arXiv  Detail & Related papers  (2025-09-29T17:45:48Z)
• MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation [55.37355146924576]
  MedSeqFT is a sequential fine-tuning framework for medical image analysis.<n>It adapts pre-trained models to new tasks while refining their representational capacity.<n>It consistently outperforms state-of-the-art fine-tuning strategies.
  arXiv  Detail & Related papers  (2025-09-07T15:22:53Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• Solving Inverse Problems with FLAIR [68.87167940623318]
  We present FLAIR, a training-free variational framework that leverages flow-based generative models as prior for inverse problems.<n>Results on standard imaging benchmarks demonstrate that FLAIR consistently outperforms existing diffusion- and flow-based methods in terms of reconstruction quality and sample diversity.
  arXiv  Detail & Related papers  (2025-06-03T09:29:47Z)
• Test-time Adaptation for Foundation Medical Segmentation Model without Parametric Updates [27.933665582178115]
  Foundation medical segmentation models, with MedSAM being the most popular, have achieved promising performance across organs and lesions.<n>MedSAM still suffers from compromised performance on specific lesions with intricate structures and appearance, as well as bounding box prompt-induced perturbations.<n>We propose to encourage maximizing factorized conditional probabilities of the posterior prediction probability using a proposed distribution-approxd latent conditional random field loss combined with an entropy minimization loss.
  arXiv  Detail & Related papers  (2025-04-02T03:03:34Z)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [67.9215891673174]
  We propose score entropy as a novel loss that naturally extends score matching to discrete spaces. We test our Score Entropy Discrete Diffusion models on standard language modeling tasks.
  arXiv  Detail & Related papers  (2023-10-25T17:59:12Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Deep Convolutional Neural Networks Predict Elasticity Tensors and their Bounds in Homogenization [0.0]
  3D convolutional neural networks (CNNs) are trained to link random heterogeneous, two-phase materials to their elastic macroscale stiffness. CNNs demonstrate the predictive accuracy not only for the standard test set but also for samples of the real, two-phase microstructure of a diamond-based coating.
  arXiv  Detail & Related papers  (2021-09-04T15:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.