The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?

• URL: http://arxiv.org/abs/2403.09037v2
• Date: Wed, 17 Jul 2024 05:54:38 GMT
• Title: The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
• Authors: Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, Stephen Gould,
• Abstract summary: This study uses linear probing to shed light on the hidden knowledge at the output layers of LVLMs. We demonstrate that the logit distributions of the first tokens contain sufficient information to determine whether to respond to the instructions.
• Score: 34.27319941609499
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs), designed to interpret and respond to human instructions, occasionally generate hallucinated or harmful content due to inappropriate instructions. This study uses linear probing to shed light on the hidden knowledge at the output layers of LVLMs. We demonstrate that the logit distributions of the first tokens contain sufficient information to determine whether to respond to the instructions, including recognizing unanswerable visual questions, defending against jailbreaking attacks, and identifying deceptive questions. Such hidden knowledge is gradually lost in logits of subsequent tokens during response generation. Then, we illustrate a simple decoding strategy at the generation of the first token, effectively improving the generated content. In experiments, we find a few interesting insights: First, the CLIP model already contains a strong signal for solving these tasks, which indicates potential bias in the existing datasets. Second, we observe performance improvement by utilizing the first logit distributions on three additional tasks, including indicating uncertainty in math solving, mitigating hallucination, and image classification. Last, with the same training data, simply finetuning LVLMs improves models' performance but is still inferior to linear probing on these tasks.

Related papers

• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models [30.26685485474035]
  Large Vision-Language Models (LVLMs) have rapidly advanced in recent years. The prevalent issue known as the hallucination' problem has emerged as a significant bottleneck. We propose a simple yet effective method named Self-Introspective Decoding (SID)
  arXiv  Detail & Related papers  (2024-08-04T13:50:17Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations. We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Conditional Prototype Rectification Prompt Learning [32.533844163120875]
  We propose a Prototype Rectification Prompt Learning (CPR) method to correct the bias of base examples and augment limited data in an effective way. CPR achieves state-of-the-art performance on both few-shot classification and base-to-new generalization tasks.
  arXiv  Detail & Related papers  (2024-04-15T15:43:52Z)
• Debiasing Multimodal Large Language Models [61.6896704217147]
  Large Vision-Language Models (LVLMs) have become indispensable tools in computer vision and natural language processing. Our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior to the input image. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners [71.8257151788923]
  We propose a novel Explainable Active Learning framework (XAL) for low-resource text classification. XAL encourages classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations. Experiments on six datasets show that XAL achieves consistent improvement over 9 strong baselines.
  arXiv  Detail & Related papers  (2023-10-09T08:07:04Z)
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding [132.78015553111234]
  Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource. We propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated.
  arXiv  Detail & Related papers  (2023-05-08T17:16:38Z)
• Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks [59.761411682238645]
  Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks. We introduce a method to incorporate evidentiality of passages -- whether a passage contains correct evidence to support the output -- into training the generator.
  arXiv  Detail & Related papers  (2021-12-16T08:18:47Z)
• Predictive Representation Learning for Language Modeling [33.08232449211759]
  Correlates of secondary information appear in LSTM representations even though they are not part of an emphexplicitly supervised prediction task. We propose Predictive Representation Learning (PRL), which explicitly constrains LSTMs to encode specific predictions.
  arXiv  Detail & Related papers  (2021-05-29T05:03:47Z)
• Predicting What You Already Know Helps: Provable Self-Supervised Learning [60.27658820909876]
  Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data. We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation. We prove the linear layer yields small approximation error even for complex ground truth function class.
  arXiv  Detail & Related papers  (2020-08-03T17:56:13Z)
• Nested Learning For Multi-Granular Tasks [24.600419295290504]
  generalize poorly to samples that are not from original training distribution. Standard deep neural networks (DNNs) are commonly trained in an end-to-end fashion for specific tasks. We introduce the concept of nested learning: how to obtain a hierarchical representation of the input. We show that nested learning outperforms the same network trained in the standard end-to-end fashion.
  arXiv  Detail & Related papers  (2020-07-13T14:27:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.