Test-Time Adaptation for Tactile-Vision-Language Models

• URL: http://arxiv.org/abs/2602.15873v1
• Date: Sat, 31 Jan 2026 02:26:01 GMT
• Title: Test-Time Adaptation for Tactile-Vision-Language Models
• Abstract summary: Tactile-vision-language (TVL) models are increasingly deployed in real-world robotic and multimodal perception tasks.<n>Existing test-time adaptation methods provide filtering in unimodal settings but lack explicit treatment of modality-wise reliability under cross-modal shifts.<n>We propose a reliability-aware framework that estimates per-modality reliability from prediction uncertainty and perturbation-based responses.
• Score: 6.918033886034668
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tactile-vision-language (TVL) models are increasingly deployed in real-world robotic and multimodal perception tasks, where test-time distribution shifts are unavoidable. Existing test-time adaptation (TTA) methods provide filtering in unimodal settings but lack explicit treatment of modality-wise reliability under asynchronous cross-modal shifts, leaving them brittle when some modalities become unreliable. We study TTA for TVL models under such shifts and propose a reliability-aware framework that estimates per-modality reliability from prediction uncertainty and perturbation-based responses. This shared reliability signal is used to (i) filter unreliable test samples, (ii) adaptively fuse tactile, visual, and language features, and (iii) regularize test-time optimization with a reliability-guided objective. On the TAG-C benchmark and additional TVL scenarios, our approach consistently outperforms strong TTA baselines, achieving accuracy gains of up to 49.9\% under severe modality corruptions, underscoring the importance of explicit modality-wise reliability modeling for robust test-time adaptation.

Related papers

• Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning [18.026220495767326]
  Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving Large Language Models (LLMs) on downstream tasks.<n>We propose Test-Time Adaptation via Stability-Aware Confidence Optimization (TASCO), a framework that incorporates local stability into confidence-based test-time adaptation.<n> Experiments demonstrate that TASCO improves reasoning accuracy and token efficiency across diverse LLMs and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-09-10T11:29:18Z)
• Robustifying Vision-Language Models via Test-Time Prompt Adaptation [23.05168102474528]
  We propose RITA, a test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment.<n>Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment.
  arXiv  Detail & Related papers  (2026-07-10T14:19:42Z)
• SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models [7.578599270855317]
  Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.<n>Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown.<n>We present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT) to evaluate the quality of each augmented view.
  arXiv  Detail & Related papers  (2026-06-05T06:12:48Z)
• On the Learnability of Test-Time Adaptation: A Recovery Complexity Perspective [25.09429306908567]
  Test-time adaptation (TTA) aims to adapt models to maintain reliable performance on non-stationary test streams without requiring labeled data.<n>Despite its empirical success, the learnability of TTA under non-stationary streams remains unexplored.<n>We propose the first theoretical framework for studying the learnability of TTA and introduce $(,)$-Recovery Complexity and $(,)$-TTA Learnability.
  arXiv  Detail & Related papers  (2026-05-27T07:04:23Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• EviATTA: Evidential Active Test-Time Adaptation for Medical Segment Anything Models [26.560028123695517]
  We propose Evidential Active Test-Time Adaptation (EviATTA), which is to our knowledge, the first ATTA framework tailored for medical SAMs.<n>Specifically, we adopt the Dirichlet-based Evidential Modeling to decompose overall predictive uncertainty into distribution uncertainty and data uncertainty.<n>We show that EviATTA consistently improves adaptation reliability with minimal expert feedback under both batch-wise and instance-wise test-time adaptation settings.
  arXiv  Detail & Related papers  (2026-03-15T23:46:31Z)
• ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models [32.840734752367275]
  Prototype-based Double-Check Separation (ProtoDCS) is a robust framework for OSTTA.<n>It separates csID and csOOD samples, enabling safe and efficient adaptation of Vision-Language Models to csID data.<n>ProtoDCS significantly boosts both known-class accuracy and OOD detection metrics.
  arXiv  Detail & Related papers  (2026-02-27T03:39:02Z)
• Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Models [10.45965859391796]
  Test-Time Adaptation (TTA) aims to improve robustness using only unlabeled test samples.<n>Most prompt-based TTA methods rely on entropy minimization.<n>We propose Fair Context Learning (FCL) that avoids entropy minimization by explicitly addressing shared-evidence bias.
  arXiv  Detail & Related papers  (2026-02-02T16:02:50Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• Uncertainty-driven Embedding Convolution [16.523816971857787]
  We propose Uncertainty-driven Embedding Convolution (UEC)<n>UEC transforms deterministic embeddings into probabilistic ones in a post-hoc manner.<n>It then computes adaptive ensemble weights based on embedding uncertainty.
  arXiv  Detail & Related papers  (2025-07-28T11:15:25Z)
• Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations [67.35596444651037]
  Vision-language models (VLMs) exhibit remarkable zero-shot capabilities but struggle with distribution shifts in downstream tasks when labeled data is unavailable.<n>We propose a Reliable Test-time Adaptation (ReTA) method that enhances reliability from two perspectives.
  arXiv  Detail & Related papers  (2025-07-13T05:37:33Z)
• TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Data [62.22804234013273]
  We propose a simple failure detection framework to unify and facilitate classification with rejection under both covariate and semantic shifts.<n>Our key insight is that by separating and consolidating failure-specific reliability knowledge with low-rank adapters, we can enhance the failure detection ability effectively and flexibly.
  arXiv  Detail & Related papers  (2025-04-20T09:20:55Z)
• UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation [93.38604803625294]
  We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) We use Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks. UncertaintyRAG outperforms baselines by 2.03% on LLaMA-2-7B, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2024-10-03T17:39:38Z)
• MedBN: Robust Test-Time Adaptation against Malicious Test Samples [11.397666167665484]
  Test-time adaptation (TTA) has emerged as a promising solution to address performance decay due to unforeseen distribution shifts between training and test data. Previous studies have uncovered security vulnerabilities within TTA even when a small proportion of the test batch is maliciously manipulated. We propose median batch normalization (MedBN), leveraging the robustness of the median for statistics estimation within the batch normalization layer during test-time inference.
  arXiv  Detail & Related papers  (2024-03-28T11:33:02Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation [1.4530711901349282]
  We propose to validate test-time adaptation methods using datasets for autonomous driving, namely CLAD-C and SHIFT. We observe that current test-time adaptation methods struggle to effectively handle varying degrees of domain shift. We enhance the well-established self-training framework by incorporating a small memory buffer to increase model stability.
  arXiv  Detail & Related papers  (2023-09-18T19:34:23Z)
• Reliable Federated Disentangling Network for Non-IID Domain Feature [62.73267904147804]
  In this paper, we propose a novel reliable federated disentangling network, termed RFedDis. To the best of our knowledge, our proposed RFedDis is the first work to develop an FL approach based on evidential uncertainty combined with feature disentangling. Our proposed RFedDis provides outstanding performance with a high degree of reliability as compared to other state-of-the-art FL approaches.
  arXiv  Detail & Related papers  (2023-01-30T11:46:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.