Zero-shot Audio Topic Reranking using Large Language Models

• URL: http://arxiv.org/abs/2309.07606v2
• Date: Tue, 10 Sep 2024 13:39:28 GMT
• Title: Zero-shot Audio Topic Reranking using Large Language Models
• Authors: Mengjie Qian, Rao Ma, Adian Liusie, Erfan Loweimi, Kate M. Knill, Mark J. F. Gales,
• Abstract summary: Multimodal Video Search by Examples (MVSE) investigates using video clips as the query term for information retrieval. This work aims to compensate for any performance loss from this rapid archive search by examining reranking approaches. Performance is evaluated for topic-based retrieval on a publicly available video archive, the BBC Rewind corpus.
• Score: 42.774019015099704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Video Search by Examples (MVSE) investigates using video clips as the query term for information retrieval, rather than the more traditional text query. This enables far richer search modalities such as images, speaker, content, topic, and emotion. A key element for this process is highly rapid and flexible search to support large archives, which in MVSE is facilitated by representing video attributes with embeddings. This work aims to compensate for any performance loss from this rapid archive search by examining reranking approaches. In particular, zero-shot reranking methods using large language models (LLMs) are investigated as these are applicable to any video archive audio content. Performance is evaluated for topic-based retrieval on a publicly available video archive, the BBC Rewind corpus. Results demonstrate that reranking significantly improves retrieval ranking without requiring any task-specific in-domain training data. Furthermore, three sources of information (ASR transcriptions, automatic summaries and synopses) as input for LLM reranking were compared. To gain a deeper understanding and further insights into the performance differences and limitations of these text sources, we employ a fact-checking approach to analyse the information consistency among them.

Related papers

• MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval [57.891157692501345]
  $textbfMultiVENT 2.0$ is a large-scale, multilingual event-centric video retrieval benchmark. It features a collection of more than 218,000 news videos and 3,906 queries targeting specific world events. Preliminary results show that state-of-the-art vision-language models struggle significantly with this task.
  arXiv  Detail & Related papers  (2024-10-15T13:56:34Z)
• Audio Captioning via Generative Pair-to-Pair Retrieval with Refined Knowledge Base [0.0]
  Retrieval-Augmented Generation (RAG) retrieves audio-text pairs from a knowledge base and augments them with query audio to generate accurate textual responses. We propose generative pair-to-pair retrieval, which uses the generated caption as a text query to accurately find relevant audio-text pairs. Our approach achieves state-of-the-art results on benchmarks including AudioCaps, Clotho, and Auto-ACD.
  arXiv  Detail & Related papers  (2024-10-14T04:57:32Z)
• GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval [56.610806615527885]
  This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX.
  arXiv  Detail & Related papers  (2024-08-14T01:24:09Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• End-to-end Knowledge Retrieval with Multi-modal Queries [50.01264794081951]
  ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model ReViz'' that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion. We demonstrate superior performance in retrieval on two datasets under zero-shot settings.
  arXiv  Detail & Related papers  (2023-06-01T08:04:12Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)
• Deep Learning for Video-Text Retrieval: a Review [13.341694455581363]
  Video-Text Retrieval (VTR) aims to search for the most relevant video related to the semantics in a given sentence. In this survey, we review and summarize over 100 research papers related to VTR.
  arXiv  Detail & Related papers  (2023-02-24T10:14:35Z)
• A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus [31.387948069111893]
  We show how to identify a short segment in a long video that semantically matches a text query. To tackle this problem, we propose the HierArchical Multi-Modal EncodeR (HAMMER) that encodes a video at both the coarse-grained clip level and the fine-trimmed frame level. We conduct extensive experiments to evaluate our model on moment localization in video corpus on ActivityNet Captions and TVR datasets.
  arXiv  Detail & Related papers  (2020-11-18T02:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.