Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection

• URL: http://arxiv.org/abs/2604.21728v2
• Date: Mon, 27 Apr 2026 13:48:45 GMT
• Title: Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection
• Abstract summary: We present Ramen, a framework for robust test-time adaptation through active sample selection.<n>For each incoming test sample, Ramen retrieves a customized batch of relevant samples from previously seen data.<n>Experiments on multiple image corruption and domain-shift benchmarks demonstrate that Ramen achieves strong and consistent performance.
• Score: 45.20212930761406
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained vision-language models such as CLIP exhibit strong zero-shot generalization but remain sensitive to distribution shifts. Test-time adaptation adapts models during inference without access to source data or target labels, offering a practical way to handle such shifts. However, existing methods typically assume that test samples come from a single, consistent domain, while in practice, test data often include samples from mixed domains with distinct characteristics. Consequently, their performance degrades under mixed-domain settings. To address this, we present Ramen, a framework for robust test-time adaptation through active sample selection. For each incoming test sample, Ramen retrieves a customized batch of relevant samples from previously seen data based on two criteria: domain consistency, which ensures that adaptation focuses on data from similar domains, and prediction balance, which mitigates adaptation bias caused by skewed predictions. To improve efficiency, Ramen employs an embedding-gradient cache that stores the embeddings and sample-level gradients of past test images. The stored embeddings are used to retrieve relevant samples, and the corresponding gradients are aggregated for model updates, eliminating the need for any additional forward or backward passes. Our theoretical analysis provides insight into why the proposed adaptation mechanism is effective under mixed-domain shifts. Experiments on multiple image corruption and domain-shift benchmarks demonstrate that Ramen achieves strong and consistent performance, offering robust and efficient adaptation in complex mixed-domain scenarios. Our code is available at https://github.com/baowenxuan/Ramen .

Related papers

• Assessing Sample Quality in Conditional Generation under Compositional Shift [34.24064416169904]
  We develop a per-sample trust score for assessing conditional samples using only the training distribution.<n>We show that the score can recover meaningful comparisons across extrapolated generations.<n>In biological imaging, selected samples preserve real morphological structure better and improve downstream predictive performance.
  arXiv  Detail & Related papers  (2026-06-08T15:10:25Z)
• Back to Source: Open-Set Continual Test-Time Adaptation via Domain Compensation [18.70132691409063]
  Test-Time Adaptation aims to mitigate distributional shifts between training and test domains during inference time.<n>The coupling of domain and semantic shifts often collapses the feature space, severely degrading both classification and out-of-distribution detection.<n>We propose DOmain COmpensation (DOCO), a lightweight and effective framework that robustly performs domain adaptation and OOD detection in a synergistic, closed loop.
  arXiv  Detail & Related papers  (2026-04-23T15:29:29Z)
• Test-Time Adaptive Object Detection with Foundation Model [39.96498227103011]
  We propose the first foundation model-powered test-time adaptive object detection method.<n>We design a Multi-modal Prompt-based Mean-Teacher framework for vision-language detector-driven test-time adaptation.<n>To guarantee high-quality pseudo-labels in every test batch, we maintain an Instance Dynamic Memory.
  arXiv  Detail & Related papers  (2025-10-29T05:19:38Z)
• PCSR: Pseudo-label Consistency-Guided Sample Refinement for Noisy Correspondence Learning [17.302186298424836]
  Cross-modal retrieval aims to align different modalities via semantic similarity.<n>Existing methods often assume that image-text pairs are perfectly aligned, overlooking Noisy Correspondences in real data.
  arXiv  Detail & Related papers  (2025-09-19T05:41:17Z)
• DOTA: Distributional Test-Time Adaptation of Vision-Language Models [69.41389326333771]
  Vision-language foundation models can be unreliable when significant distribution gaps exist between training and test data.<n>We propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation.<n>This distribution-centric approach enables the model to continually learn and adapt to the deployment environment.
  arXiv  Detail & Related papers  (2024-09-28T15:03:28Z)
• Resilient Practical Test-Time Adaptation: Soft Batch Normalization Alignment and Entropy-driven Memory Bank [24.096250529224914]
  We propose a practical test-time adaptation (ResiTTA) method focused on parameter resilience and data quality. We use an entropy-driven memory bank that accounts for timeliness, the persistence of over-confident samples, and sample uncertainty for high-quality data in adaptation. We empirically validate ResiTTA across various benchmark datasets, demonstrating state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-01-26T03:24:55Z)
• Dual Adaptive Representation Alignment for Cross-domain Few-shot Learning [58.837146720228226]
  Few-shot learning aims to recognize novel queries with limited support samples by learning from base knowledge. Recent progress in this setting assumes that the base knowledge and novel query samples are distributed in the same domains. We propose to address the cross-domain few-shot learning problem where only extremely few samples are available in target domains.
  arXiv  Detail & Related papers  (2023-06-18T09:52:16Z)
• Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition [65.84978547406753]
  Test-time Adaptation aims to adapt the model trained on source domains to yield better predictions for test samples. Single-Utterance Test-time Adaptation (SUTA) is the first TTA study in speech area to our best knowledge.
  arXiv  Detail & Related papers  (2022-03-27T06:38:39Z)
• MEMO: Test Time Robustness via Adaptation and Augmentation [131.28104376280197]
  We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions. We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
  arXiv  Detail & Related papers  (2021-10-18T17:55:11Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.