D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models

• URL: http://arxiv.org/abs/2510.09473v2
• Date: Fri, 17 Oct 2025 01:32:13 GMT
• Title: D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models
• Abstract summary: Test-time adaptation paradigm provides flexibility towards domain shifts.<n>Vision-Language Models (VLMs) leverage their generalization capabilities for diverse downstream tasks.
• Score: 5.770351255180494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time adaptation paradigm provides flexibility towards domain shifts by performing immediate adaptation on unlabeled target data from the source model. Vision-Language Models (VLMs) leverage their generalization capabilities for diverse downstream tasks, and test-time prompt tuning has emerged as a prominent solution for adapting VLMs. In this work, we explore contrastive VLMs and identify the modality gap caused by a single dominant feature dimension across modalities. We observe that the dominant dimensions in both text and image modalities exhibit high predictive sensitivity, and that constraining their influence can improve calibration error. Building on this insight, we propose dimensional entropy maximization that regularizes the distribution of textual features toward uniformity to mitigate the dependency of dominant dimensions. Our method alleviates the degradation of calibration performance in test-time prompt tuning, offering a simple yet effective solution to enhance the reliability of VLMs in real-world deployment scenarios.

Related papers

• From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model [81.67708143891319]
  Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.<n>We bridge the detached objectives of inference and adaptation forVLMs, and propose a principled VLM TTA method called algname.<n>Experiments demonstrate the effectiveness and efficiency of our methods, outperforming the best-performing methods by up to 7% with state-of-the-art efficiency.
  arXiv  Detail & Related papers  (2026-08-18T21:44:32Z)
• Prompt-Driven Simulation with Feature Perturbation for Cross-Domain Few-Shot Object Detection [8.13477193884434]
  PSP-FSOD is a principled framework that integrates prompt-driven domain simulation with feature perturbation regularization.<n> PSP-FSOD produces high-quality domain-diverse supervision and learns domain-invariant representations.
  arXiv  Detail & Related papers  (2026-08-02T16:11:16Z)
• Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection [62.89126207012712]
  Face Presentation Attack Detection (PAD) demands incremental learning to combat spoofing tactics and domains.<n>Privacy regulations forbid retaining past data, necessitating rehearsal-free learning (RF-IL)
  arXiv  Detail & Related papers  (2025-12-22T04:30:11Z)
• A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models [19.257897956175814]
  Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data.<n>We propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features.<n>We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error.
  arXiv  Detail & Related papers  (2025-10-30T12:45:24Z)
• Unified modality separation: A vision-language framework for unsupervised domain adaptation [60.8391821117794]
  Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains.<n>We propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components.<n>Our methods achieve up to 9% performance gain with 9 times of computational efficiencies.
  arXiv  Detail & Related papers  (2025-08-07T02:51:10Z)
• Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations [67.35596444651037]
  Vision-language models (VLMs) exhibit remarkable zero-shot capabilities but struggle with distribution shifts in downstream tasks when labeled data is unavailable.<n>We propose a Reliable Test-time Adaptation (ReTA) method that enhances reliability from two perspectives.
  arXiv  Detail & Related papers  (2025-07-13T05:37:33Z)
• Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation [26.580361841501514]
  Vision-language models (VLMs) excel in various multimodal tasks but frequently suffer from poor calibration.<n>This miscalibration undermines user trust, especially when models confidently provide incorrect or fabricated information.<n>We propose a novel Confidence through Semantic Perturbation (CSP) framework to improve the calibration of verbalized confidence for object-centric queries.
  arXiv  Detail & Related papers  (2025-04-21T04:01:22Z)
• O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models [17.56932003351322]
  Test-time prompt tuning for vision-language models (VLMs) is getting attention because of their ability to learn with unlabeled data without fine-tuning.<n>The resulting models tend to demonstrate poor calibration, which casts doubts on the reliability and trustworthiness of these models.<n>We propose a new approach, called O-TPT, that introduces orthogonality constraints on the textual features corresponding to the learnable prompts.
  arXiv  Detail & Related papers  (2025-03-15T11:45:54Z)
• Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
  Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. We propose a test-time Forward-Optimization Adaptation (FOA) method. FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
  arXiv  Detail & Related papers  (2024-04-02T05:34:33Z)
• Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer [54.32283739486781]
  We present a textbfForgery-aware textbfAdaptive textbfVision textbfTransformer (FA-ViT) under the adaptive learning paradigm. FA-ViT achieves 93.83% and 78.32% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation.
  arXiv  Detail & Related papers  (2023-09-20T06:51:11Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypes [18.140619966865955]
  We propose a novel test-time adaptation strategy that adjusts the model pre-trained on the source domain using only unlabeled online data from the target domain. We show that our method exhibits state-of-the-art performance on various standard benchmarks and even outperforms its supervised counterpart.
  arXiv  Detail & Related papers  (2022-07-24T10:17:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.