Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

• URL: http://arxiv.org/abs/2604.09532v1
• Date: Fri, 10 Apr 2026 17:48:56 GMT
• Title: Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
• Abstract summary: We propose VisPrompt, a vision-guided prompt learning framework for noisy-label settings.<n>We exploit a cross-modal attention mechanism to reversely inject visual semantics into prompt representations.<n>VisPrompt significantly improves robustness while keeping the pretrained VLM backbone frozen and introducing only a small amount of additional trainable parameters.
• Score: 19.372722047131862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt learning is a parameter-efficient approach for vision-language models, yet its robustness under label noise is less investigated. Visual content contains richer and more reliable semantic information, which remains more robust under label noise. However, the prompt itself is highly susceptible to label noise. Motivated by this intuition, we propose VisPrompt, a lightweight and robust vision-guided prompt learning framework for noisy-label settings. Specifically, we exploit a cross-modal attention mechanism to reversely inject visual semantics into prompt representations. This enables the prompt tokens to selectively aggregate visual information relevant to the current sample, thereby improving robustness by anchoring prompt learning to stable instance-level visual evidence and reducing the influence of noisy supervision. To address the instability caused by using the same way of injecting visual information for all samples, despite differences in the quality of their visual cues, we further introduce a lightweight conditional modulation mechanism to adaptively control the strength of visual information injection, which strikes a more robust balance between text-side semantic priors and image-side instance evidence. The proposed framework effectively suppresses the noise-induced disturbances, reduce instability in prompt updates, and alleviate memorization of mislabeled samples. VisPrompt significantly improves robustness while keeping the pretrained VLM backbone frozen and introducing only a small amount of additional trainable parameters. Extensive experiments under synthetic and real-world label noise demonstrate that VisPrompt generally outperforms existing baselines on seven benchmark datasets and achieves stronger robustness. Our code is publicly available at https://github.com/gezbww/Vis_Prompt.

Related papers

• Information-Regularized Attention for Visual-Centric Reasoning [30.245558374164492]
  Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting.<n>We introduce Information-Regularized Attention (IRA), a attention mechanism that explicitly regulates the amount of visual information injected into the hidden states of intermediate transformer layers.
  arXiv  Detail & Related papers  (2026-07-01T04:45:41Z)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation [59.80828456543202]
  We propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation.<n>This mechanism requires only minimal training for calibration and can be applied across diverse vision-language models.<n>We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.
  arXiv  Detail & Related papers  (2026-06-02T08:23:28Z)
• Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning [79.61386774847357]
  Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts.<n>The choice of such prompts largely influences VICL performance, standing out as a key challenge.<n>We develop a framework named LaPR (Label-aware Prompt Retrieval) which highlights the role of labels in prompt selection.
  arXiv  Detail & Related papers  (2026-04-04T09:18:31Z)
• Is the Information Bottleneck Robust Enough? Towards Label-Noise Resistant Information Bottleneck Learning [32.856351223184696]
  The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information.<n>We propose LaT-IB, a novel Label-Noise ResistanT Information Bottleneck method which introduces a "Minimal-Sufficient-Clean" (MSC) criterion.<n>LaT-IB achieves superior robustness and efficiency under label noise, significantly enhancing robustness and applicability in real-world scenarios with label noise.
  arXiv  Detail & Related papers  (2025-12-11T12:01:20Z)
• Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise [9.536089523962486]
  We propose ANPrompt, a robust prompt tuning framework that actively incorporates weak semantic noise.<n>We show that ANPrompt consistently outperforms existing prompt tuning methods.<n>It offers superior robustness to semantic noise and improved generalization across tasks.
  arXiv  Detail & Related papers  (2025-08-06T17:42:30Z)
• NLPrompt: Noise-Label Prompt Learning for Vision-Language Models [45.35555264802591]
  Real-world datasets often contain noisy labels that can degrade prompt learning performance.<n>In this paper, we demonstrate that using mean absolute error (MAE) loss in prompt learning, named PromptMAE, significantly enhances robustness against noisy labels.<n>We also introduce PromptOT, a prompt-based optimal transport data purification method to enhance the robustness further.
  arXiv  Detail & Related papers  (2024-12-02T08:25:09Z)
• KNN Transformer with Pyramid Prompts for Few-Shot Learning [52.735070934075736]
  Few-Shot Learning aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features.
  arXiv  Detail & Related papers  (2024-10-14T07:39:30Z)
• Vision-Language Models are Strong Noisy Label Detectors [76.07846780815794]
  This paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.
  arXiv  Detail & Related papers  (2024-09-29T12:55:17Z)
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis [69.48582264712854]
  We propose a robust learning method to perform robust visual sentiment analysis. Our method relies on an external memory to aggregate and filter noisy labels during training. We establish a benchmark for visual sentiment analysis with label noise using publicly available datasets.
  arXiv  Detail & Related papers  (2021-09-15T18:18:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.