ViMU: Benchmarking Video Metaphorical Understanding

• URL: http://arxiv.org/abs/2605.14607v1
• Date: Thu, 14 May 2026 09:23:59 GMT
• Title: ViMU: Benchmarking Video Metaphorical Understanding
• Abstract summary: ViMU is a benchmark designed to evaluate the subtext understanding capabilities of frontier models in videos.<n>It assesses whether video understanding models can go beyond literal perception to infer implicit meaning.<n>All questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering.
• Score: 58.432996881401415
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Any new medium, once it emerges, is used for more than the transmission of overt content alone. The information it carries typically operates on two levels: one is the content directly presented, while the other is the subtext beneath it-the implicit ideas and intentions the creator seeks to convey through the medium. Likewise, since video technologies became widely adopted, video has served not only as a powerful tool for recording and communicating visual information, but also as a vehicle for emotions, attitudes, and social meanings that are often difficult to articulate explicitly. Thus, the true meaning of many videos does not reside solely in what is shown on screen; it is often embedded in context, style of expression, and the viewer's social experience. Some forms of such video subtext are humorous, while others carry irony, mockery, or criticism. These implicit meanings can also be interpreted very differently across cultural backgrounds and social groups. However, most existing video understanding models still focus primarily on literal visual comprehension, such as recognizing objects, actions, or temporal relations, and lack a systematic ability to understand the metaphorical, ironic, and social meanings embedded in videos. To bridge this gap, we introduce ViMU, the first benchmark designed to systematically evaluate the subtext understanding capabilities of frontier models in videos. ViMU assesses whether video understanding models can go beyond literal perception to infer implicit meaning while grounding their interpretations in multimodal evidence and answering both open-ended and multiple-choice questions. Importantly, all questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering.

Related papers

• Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos [24.604118147581886]
  We evaluate whether models can infer the implicit, non-linear, and rhetorically layered meanings of social media videos.<n>DrivelHub+ consists of 1,000 videos collected from social media, each annotated with a human-written implicit narrative explanation.
  arXiv  Detail & Related papers  (2026-08-05T15:06:03Z)
• MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models [56.49314029765706]
  We introduce MESH, a benchmark designed to evaluate hallucinations in LVMs systematically.<n>MESH uses a Question-Answering framework with binary and multi-choice formats incorporating target and trap instances.<n>We demonstrate that MESH offers an effective and comprehensive approach for identifying hallucinations in videos.
  arXiv  Detail & Related papers  (2025-09-10T12:34:07Z)
• All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark [70.92907745196153]
  MAIA is a benchmark for fine-grained investigation of the reasoning abilities of visual language models on videos.<n>It considers twelve reasoning categories that aim to disentangle language and vision relations by highlighting the role of the visual input.<n>MAIA differs from other available video benchmarks for its design, its reasoning categories, the metric it uses, and the language and culture of the videos.
  arXiv  Detail & Related papers  (2025-02-24T09:25:51Z)
• Unveiling the Invisible: Captioning Videos with Metaphors [43.53477124719281]
  We introduce a new Vision-Language (VL) task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a dataset with 705 videos and 2115 human-written captions. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task.
  arXiv  Detail & Related papers  (2024-06-07T12:32:44Z)
• HENASY: Learning to Assemble Scene-Entities for Egocentric Video-Language Model [9.762722976833581]
  Current models rely extensively on instance-level alignment between video and language modalities. We take an inspiration from human perception and explore a compositional approach for ego video representation.
  arXiv  Detail & Related papers  (2024-06-01T05:41:12Z)
• VideoDistill: Language-aware Vision Distillation for Video Question Answering [24.675876324457747]
  We propose VideoDistill, a framework with language-aware (i.e., goal-driven) behavior in both vision perception and answer generation process. VideoDistill generates answers only from question-related visual embeddings. We conduct experimental evaluations on various challenging video question-answering benchmarks, and VideoDistill achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-01T07:44:24Z)
• A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot [67.00455874279383]
  We propose verbalizing long videos to generate descriptions in natural language, then performing video-understanding tasks on the generated story as opposed to the original video. Our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. To alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.
  arXiv  Detail & Related papers  (2023-05-16T19:13:11Z)
• How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios [73.24092762346095]
  We introduce two large-scale datasets with over 60,000 videos annotated for emotional response and subjective wellbeing. The Video Cognitive Empathy dataset contains annotations for distributions of fine-grained emotional responses, allowing models to gain a detailed understanding of affective states. The Video to Valence dataset contains annotations of relative pleasantness between videos, which enables predicting a continuous spectrum of wellbeing.
  arXiv  Detail & Related papers  (2022-10-18T17:58:25Z)
• Video as Conditional Graph Hierarchy for Multi-Granular Question Answering [80.94367625007352]
  We argue that while video is presented in frame sequence, the visual elements are not sequential but rather hierarchical in semantic space. We propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner.
  arXiv  Detail & Related papers  (2021-12-12T10:35:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.