Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos

• URL: http://arxiv.org/abs/2608.04939v1
• Date: Wed, 05 Aug 2026 15:06:03 GMT
• Title: Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos
• Abstract summary: We evaluate whether models can infer the implicit, non-linear, and rhetorically layered meanings of social media videos.<n>DrivelHub+ consists of 1,000 videos collected from social media, each annotated with a human-written implicit narrative explanation.
• Score: 24.604118147581886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Social media videos often communicate meanings that go beyond their visible actions, captions, or speech. A mundane clip may become humorous, ironic, or satire only through the interaction of multimodal cues and cultural context, making such content a difficult test case for video-language models. In this paper, we introduce \textit{DrivelHub+}, a benchmark for evaluating whether models can infer the implicit, non-linear, and rhetorically layered meanings of social media videos that appear nonsensical on the surface but convey deliberate pragmatic meanings. DrivelHub+ consists of 1,000 videos collected from social media, each annotated with a human-written implicit narrative explanation. Unlike conventional video understanding tasks focused on recognition or description, we present a benchmark that targets contextual multimodal reasoning. We evaluate current video-language models from two perspectives: explanation, where models must explain the pragmatic comprehension of a video in natural language; and representation, where we adapt reasoning-as-retrieval to test whether model representations align videos with their corresponding implicit narratives in both video-to-text and text-to-video retrieval. Our benchmark provides a diagnostic setting for measuring the gap between multimodal perception and pragmatic comprehension, asking whether current models can move beyond describing what is shown to inferring what is meant.

Related papers

• ViMU: Benchmarking Video Metaphorical Understanding [58.432996881401415]
  ViMU is a benchmark designed to evaluate the subtext understanding capabilities of frontier models in videos.<n>It assesses whether video understanding models can go beyond literal perception to infer implicit meaning.<n>All questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering.
  arXiv  Detail & Related papers  (2026-05-14T09:23:59Z)
• HENASY: Learning to Assemble Scene-Entities for Egocentric Video-Language Model [9.762722976833581]
  Current models rely extensively on instance-level alignment between video and language modalities. We take an inspiration from human perception and explore a compositional approach for ego video representation.
  arXiv  Detail & Related papers  (2024-06-01T05:41:12Z)
• SPOT! Revisiting Video-Language Models for Event Understanding [31.49859545456809]
  We introduce SPOT Prober, to benchmark existing video-language models's capacities of distinguishing event-level discrepancies. We evaluate the existing video-language models with these positive and negative captions and find they fail to distinguish most of the manipulated events. Based on our findings, we propose to plug in these manipulated event captions as hard negative samples and find them effective in enhancing models for event understanding.
  arXiv  Detail & Related papers  (2023-11-21T18:43:07Z)
• Video as Conditional Graph Hierarchy for Multi-Granular Question Answering [80.94367625007352]
  We argue that while video is presented in frame sequence, the visual elements are not sequential but rather hierarchical in semantic space. We propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner.
  arXiv  Detail & Related papers  (2021-12-12T10:35:19Z)
• Cross-Modal Graph with Meta Concepts for Video Captioning [101.97397967958722]
  We propose Cross-Modal Graph (CMG) with meta concepts for video captioning. To cover the useful semantic concepts in video captions, we weakly learn the corresponding visual regions for text descriptions. We construct holistic video-level and local frame-level video graphs with the predicted predicates to model video sequence structures.
  arXiv  Detail & Related papers  (2021-08-14T04:00:42Z)
• Video SemNet: Memory-Augmented Video Semantic Network [14.64546899992196]
  We propose a machine learning approach to capture the narrative elements in movies by bridging the gap between the low-level data representations and semantic aspects of the visual medium. We present a Memory-Augmented Video Semantic Network, called Video SemNet, to encode the semantic descriptors and learn an embedding for the video. We demonstrate that our model is able to predict genres and IMDB ratings with a weighted F-1 score of 0.72 and 0.63 respectively.
  arXiv  Detail & Related papers  (2020-11-22T01:36:37Z)
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision [54.73758942064708]
  We teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations. For training and evaluation, we contribute a new dataset ApartmenTour' that contains a large number of online videos and subtitles.
  arXiv  Detail & Related papers  (2020-11-19T03:43:56Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference [103.7457132841367]
  We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given a video clip with subtitles aligned as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by the given video clip. A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips.
  arXiv  Detail & Related papers  (2020-03-25T20:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.