Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models

• URL: http://arxiv.org/abs/2406.08402v1
• Date: Wed, 12 Jun 2024 16:51:54 GMT
• Title: Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models
• Authors: Chun-Yi Kuan, Wei-Ping Huang, Hung-yi Lee,
• Abstract summary: We introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content. We explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions.
• Score: 49.87432626548563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks. Previous research has primarily focused on assessing the performance of LALMs across various tasks, yet overlooking their reliability, particularly concerning issues like object hallucination. In our study, we introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip. This limitation highlights a critical weakness in current LALMs: their inadequate understanding of discriminative queries. Moreover, we explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions.

Related papers

• AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models [27.430040932849018]
  We introduce AVHBench, the first comprehensive benchmark specifically designed to evaluate the perception and comprehension capabilities of audio-visual models. Our results reveal that most existing audio-visual LLMs struggle with hallucinations caused by cross-interactions between modalities. Simple training with our AVHBench improves robustness of audio-visual LLMs against hallucinations.
  arXiv  Detail & Related papers  (2024-10-23T23:36:06Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information. These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)
• Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models [0.9285295512807729]
  Audio Question Answering has garnered attention due to the advent of Large Audio Language Models. While LALMs excel in general audio understanding, they are limited in temporal reasoning. This paper addresses these challenges and limitations in audio temporal reasoning.
  arXiv  Detail & Related papers  (2024-09-10T05:26:53Z)
• Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time [73.7845280328535]
  We present Meerkat, an audio-visual LLM equipped with a fine-grained understanding of image and audio. Meerkat can tackle challenging tasks such as audio referred image grounding, image guided audio temporal localization, and audio-visual fact-checking. We achieve state-of-the-art performance on all these downstream tasks with a relative improvement of up to 37.12%.
  arXiv  Detail & Related papers  (2024-07-01T23:32:25Z)
• Hallucination of Multimodal Large Language Models: A Survey [40.73148186369018]
  multimodal large language models (MLLMs) have demonstrated significant advancements and remarkable abilities in multimodal tasks. Despite these promising developments, MLLMs often generate outputs that are inconsistent with the visual content. This survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field.
  arXiv  Detail & Related papers  (2024-04-29T17:59:41Z)
• Exploring Perceptual Limitation of Multimodal Large Language Models [57.567868157293994]
  We quantitatively study the perception of small visual objects in several state-of-the-art MLLMs. We identify four independent factors that can contribute to this limitation. Lower object quality and smaller object size can both independently reduce MLLMs' ability to answer visual questions.
  arXiv  Detail & Related papers  (2024-02-12T03:04:42Z)
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models [116.01843550398183]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of downstream tasks. LLMs occasionally generate content that diverges from the user input, contradicts previously generated context, or misaligns with established world knowledge.
  arXiv  Detail & Related papers  (2023-09-03T16:56:48Z)
• Audio Self-supervised Learning: A Survey [60.41768569891083]
  Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing.
  arXiv  Detail & Related papers  (2022-03-02T15:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.