Unveiling the Invisible: Captioning Videos with Metaphors

• URL: http://arxiv.org/abs/2406.04886v2
• Date: Wed, 02 Oct 2024 13:40:10 GMT
• Title: Unveiling the Invisible: Captioning Videos with Metaphors
• Authors: Abisek Rajakumar Kalarani, Pushpak Bhattacharyya, Sumit Shekhar,
• Abstract summary: We introduce a new Vision-Language (VL) task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a dataset with 705 videos and 2115 human-written captions. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task.
• Score: 43.53477124719281
• License:
• Abstract: Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.

Related papers

• A framework for annotating and modelling intentions behind metaphor use [12.40493670580608]
  We propose a novel taxonomy of intentions commonly attributed to metaphor, which comprises 9 categories. We also release the first dataset annotated for intentions behind metaphor use. We use this dataset to test the capability of large language models (LLMs) in inferring the intentions behind metaphor use, in zero- and in-context few-shot settings.
  arXiv  Detail & Related papers  (2024-07-04T14:13:57Z)
• I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors [38.70166865926743]
  We propose a new task of generating visual metaphors from linguistic metaphors. This is a challenging task for diffusion-based text-to-image models, since it requires the ability to model implicit meaning and compositionality. We create a high-quality dataset containing 6,476 visual metaphors for 1,540 linguistic metaphors and their associated visual elaborations.
  arXiv  Detail & Related papers  (2023-05-24T05:01:10Z)
• Fine-grained Audible Video Description [61.81122862375985]
  We construct the first fine-grained audible video description benchmark (FAVDBench) For each video clip, we first provide a one-sentence summary of the video, followed by 4-6 sentences describing the visual details and 1-2 audio-related descriptions at the end. We demonstrate that employing fine-grained video descriptions can create more intricate videos than using captions.
  arXiv  Detail & Related papers  (2023-03-27T22:03:48Z)
• MetaCLUE: Towards Comprehensive Visual Metaphors Research [43.604408485890275]
  We introduce MetaCLUE, a set of vision tasks on visual metaphor. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.
  arXiv  Detail & Related papers  (2022-12-19T22:41:46Z)
• Cross-Modal Graph with Meta Concepts for Video Captioning [101.97397967958722]
  We propose Cross-Modal Graph (CMG) with meta concepts for video captioning. To cover the useful semantic concepts in video captions, we weakly learn the corresponding visual regions for text descriptions. We construct holistic video-level and local frame-level video graphs with the predicted predicates to model video sequence structures.
  arXiv  Detail & Related papers  (2021-08-14T04:00:42Z)
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision [54.73758942064708]
  We teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations. For training and evaluation, we contribute a new dataset ApartmenTour' that contains a large number of online videos and subtitles.
  arXiv  Detail & Related papers  (2020-11-19T03:43:56Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• Analogical Reasoning for Visually Grounded Language Acquisition [55.14286413675306]
  Children acquire language subconsciously by observing the surrounding world and listening to descriptions. In this paper, we bring this ability to AI, by studying the task of Visually grounded Language Acquisition. We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning.
  arXiv  Detail & Related papers  (2020-07-22T20:51:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.