USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

• URL: http://arxiv.org/abs/2607.03900v1
• Date: Sat, 04 Jul 2026 14:45:23 GMT
• Title: USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
• Abstract summary: Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models.<n>We propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages.
• Score: 54.44143638529487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.

Related papers

• TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models [49.463896453707065]
  Vision-Language-Action (VLA) models have made remarkable progress, but they remain vulnerable to distribution shifts at deployment time.<n>Recent VLA models suggest that prompts can serve as an efficient interface for steering policy behavior, but existing prompt-based steering typically relies on external guidance.<n>This raises a natural question: can test-time training (TTT) for VLA be achieved by optimizing a prompt, so that the steering interface itself can be learned and adapted from interaction?<n>We address this question with TTT-VLA, a test-time training framework based on Latent Prompt Optimization (LPO)
  arXiv  Detail & Related papers  (2026-06-02T04:10:39Z)
• Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models [96.0074341403456]
  Inference-time compute has re-emerged as a practical way to improve LLM reasoning.<n>Most test-time scaling (TTS) algorithms rely on autoregressive decoding.<n>We propose Prism, an efficient TTS framework for dLLMs.
  arXiv  Detail & Related papers  (2026-02-02T09:14:51Z)
• MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models [62.20230218401528]
  We propose Meta Test-Time Prompt Tuning (MetaTPT), a meta-learning framework that learns a self-supervised auxiliary task to guide test-time prompt tuning.<n>By coupling augmentation learning with prompt tuning, MetaTPT improves test-time adaptation under domain shifts.
  arXiv  Detail & Related papers  (2025-12-13T10:23:10Z)
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates [5.84817561920117]
  Test-Time Adaptation adapts vision-language models to unlabeled test data in new domains.<n>Current cache-based TTA models store only a limited set of high-confidence samples.<n>We propose a Recursive Updating module that integrates all incoming test samples.
  arXiv  Detail & Related papers  (2025-08-07T23:11:33Z)
• TAPS : Frustratingly Simple Test Time Active Learning for VLMs [0.0]
  Test-Time Optimization enables models to adapt to new data during inference by updating parameters on-the-fly.<n>We propose a novel Test-Time Active Learning framework that adaptively queries uncertain samples and updates prompts dynamically.<n>Our framework provides a practical and effective solution for real-world deployment in safety-critical applications such as autonomous systems and medical diagnostics.
  arXiv  Detail & Related papers  (2025-07-26T18:04:49Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling [39.57154199908565]
  Self-Enhanced Test-Time Scaling (SETS) is a simple yet effective approach that overcomes limitations by strategically combining parallel and sequential techniques.<n>SETS exploits the inherent self-verification and self- computation capabilities of Large Language Models, unifying sampling, verification, and correction within a single framework.<n>Our results show SETS achieves significant performance improvements and more advantageous test-time scaling behavior than the alternatives.
  arXiv  Detail & Related papers  (2025-01-31T17:03:16Z)
• Active Test-Time Adaptation: Theoretical Analyses and An Algorithm [51.84691955495693]
  Test-time adaptation (TTA) addresses distribution shifts for streaming test data in unsupervised settings. We propose the novel problem setting of active test-time adaptation (ATTA) that integrates active learning within the fully TTA setting.
  arXiv  Detail & Related papers  (2024-04-07T22:31:34Z)
• Few Clicks Suffice: Active Test-Time Adaptation for Semantic Segmentation [14.112999441288615]
  Test-time adaptation (TTA) adapts pre-trained models during inference using unlabeled test data. There is still a significant performance gap between the TTA approaches and their supervised counterparts. We propose ATASeg framework, which consists of two parts, i.e., model adapter and label annotator.
  arXiv  Detail & Related papers  (2023-12-04T12:16:02Z)
• Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition [30.350005654271868]
  Test-Time Adaptation aims to utilize the test stream to adjust predictions in real-time inference. High computational cost makes it intractable to run on resource-constrained edge devices. We propose an Optimization-Free Test-Time Adaptation framework for sensor-based HAR.
  arXiv  Detail & Related papers  (2023-10-28T02:20:33Z)
• Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization [101.08992036691673]
  This paper explores a realistic unsupervised fine-tuning scenario, considering the presence of out-of-distribution samples from unknown classes. In particular, we focus on simultaneously enhancing out-of-distribution detection and the recognition of instances associated with known classes. We present a simple, efficient, and effective approach called Universal Entropy Optimization (UEO)
  arXiv  Detail & Related papers  (2023-08-24T16:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.