LUST: A Multi-Modal Framework with Hierarchical LLM-based Scoring for Learned Thematic Significance Tracking in Multimedia Content

• URL: http://arxiv.org/abs/2508.04353v1
• Date: Wed, 06 Aug 2025 11:48:51 GMT
• Title: LUST: A Multi-Modal Framework with Hierarchical LLM-based Scoring for Learned Thematic Significance Tracking in Multimedia Content
• Abstract summary: The Learned User Significance Tracker (LUST) is a framework designed to analyze video content and quantify the thematic relevance of its segments.<n>The core innovation lies in a hierarchical, two-stage relevance scoring mechanism employing Large Language Models (LLMs)<n>The LUST framework aims to provide a nuanced, temporally-aware measure of user-defined significance, outputting an annotated video with visualized relevance scores and comprehensive analytical logs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces the Learned User Significance Tracker (LUST), a framework designed to analyze video content and quantify the thematic relevance of its segments in relation to a user-provided textual description of significance. LUST leverages a multi-modal analytical pipeline, integrating visual cues from video frames with textual information extracted via Automatic Speech Recognition (ASR) from the audio track. The core innovation lies in a hierarchical, two-stage relevance scoring mechanism employing Large Language Models (LLMs). An initial "direct relevance" score, $S_{d,i}$, assesses individual segments based on immediate visual and auditory content against the theme. This is followed by a "contextual relevance" score, $S_{c,i}$, that refines the assessment by incorporating the temporal progression of preceding thematic scores, allowing the model to understand evolving narratives. The LUST framework aims to provide a nuanced, temporally-aware measure of user-defined significance, outputting an annotated video with visualized relevance scores and comprehensive analytical logs.

Related papers

• Scribby: A Multi-Level LLM Framework for Semantic Video Analysis [0.0]
  This paper proposes an LLM-based video summarization framework that balances macro-level comprehension with micro-level semantic analysis cite6,12,13.<n>Contextual continuity is retained during sentence-level processing by incorporating both the global transcript analysis and adjacent sentence information into each evaluation prompt.<n>This framework establishes a foundation for video analysis tools that visualize semantic chunking and semantic matching through relevance-based heatmaps.
  arXiv  Detail & Related papers  (2026-06-08T14:41:00Z)
• Decoding the Hook: A Multimodal LLM Framework for Analyzing the Hooking Period of Video Ads [9.34170961508317]
  Video ads are a vital medium for brands to engage consumers, with social media platforms leveraging user data to optimize ad delivery and boost engagement.<n>A crucial but under-explored aspect is the 'hooking period', the first three seconds that capture viewer attention and influence engagement metrics.<n>This study presents a framework using transformer-based multimodal large language models (MLLMs) to analyze the hooking period of video ads.
  arXiv  Detail & Related papers  (2026-02-25T18:24:06Z)
• Understanding Virality: A Rubric based Vision-Language Model Framework for Short-Form Edutainment Evaluation [8.15791379444665]
  VideoScore-2 does not capture how specific audiovisual attributes drive real audience engagement.<n>We propose a data-driven evaluation framework that uses Vision-Language Models (VLMs) to extract unsupervised audiovisual features.<n>Our approach advances toward robust and explainable video understanding.
  arXiv  Detail & Related papers  (2025-12-24T19:43:59Z)
• Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction [33.79474114703357]
  We propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI)<n>Our model consists of two core components: Cross-modal Frame Aggregation and Context-aware Feature Enhancement.<n>Experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.
  arXiv  Detail & Related papers  (2025-11-13T09:48:12Z)
• DTECT: Dynamic Topic Explorer & Context Tracker [0.8962460460173959]
  We introduce DTECT (Dynamic Topic Explorer & Context Tracker), an end-to-end system that bridges the gap between raw textual data and meaningful temporal insights.<n>DTECT provides a unified workflow that supports data preprocessing, multiple model architectures, and dedicated evaluation metrics to analyze the topic quality of temporal topic models.<n>It significantly enhances interpretability by introducing LLM-driven automatic topic labeling, trend analysis via temporally salient words, interactive visualizations with document-level summarization, and a natural language chat interface for intuitive data querying.
  arXiv  Detail & Related papers  (2025-07-10T16:44:33Z)
• VidText: Towards Comprehensive Evaluation for Video Text Understanding [54.15328647518558]
  VidText is a benchmark for comprehensive and in-depth evaluation of video text understanding.<n>It covers a wide range of real-world scenarios and supports multilingual content.<n>It introduces a hierarchical evaluation framework with video-level, clip-level, and instance-level tasks.
  arXiv  Detail & Related papers  (2025-05-28T19:39:35Z)
• Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction [18.24629930062925]
  Partially Relevant Video Retrieval aims to retrieve the target video that is partially relevant to a text query.<n>Existing methods coarsely align paired videos and text queries to construct the semantic space.<n>We propose a novel PRVR framework to systematically exploit inter-sample correlation and intra-sample redundancy.
  arXiv  Detail & Related papers  (2025-04-28T09:52:46Z)
• Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning [56.873534081386]
  A new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning.<n>We propose a query-centric audio-visual cognition network to construct a reliable multi-modal representation for three tasks.<n>This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks.
  arXiv  Detail & Related papers  (2024-12-18T06:43:06Z)
• Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives [0.0]
  We propose an enhanced framework that integrates a Causal-Temporal Reasoning Module into state-of-the-art LVLMs.<n>CTRM comprises two key components: the Causal Dynamics (CDE) and the Temporal Learner (TRL)<n>We design a multi-stage learning strategy to optimize the model, combining pre-training on large-scale video-text datasets.
  arXiv  Detail & Related papers  (2024-12-14T07:28:38Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• DeVAn: Dense Video Annotation for Video-Language Models [68.70692422636313]
  We present a novel human annotated dataset for evaluating the ability for visual-language models to generate descriptions for real-world video clips. The dataset contains 8.5K YouTube video clips of 20-60 seconds in duration and covers a wide range of topics and interests.
  arXiv  Detail & Related papers  (2023-10-08T08:02:43Z)
• Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos [67.12603318660689]
  We propose a novel Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN) HVSARN enables both visual- and semantic-aware query reasoning from object-level to frame-level. Experiments on three datasets demonstrate that our HVSARN achieves a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-02T08:00:22Z)
• Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense [98.70218717851665]
  It is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. We present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation.
  arXiv  Detail & Related papers  (2022-11-10T21:44:33Z)
• A Multimodal Framework for Video Ads Understanding [64.70769354696019]
  We develop a multimodal system to improve the ability of structured analysis of advertising video content. Our solution achieved a score of 0.2470 measured in consideration of localization and prediction accuracy, ranking fourth in the 2021 TAAC final leaderboard.
  arXiv  Detail & Related papers  (2021-08-29T16:06:00Z)
• MOPT: Multi-Object Panoptic Tracking [33.77171216778909]
  We introduce a novel perception task denoted as multi-object panoptic tracking (MOPT) MOPT allows for exploiting pixel-level semantic information of 'thing' and'stuff' classes, temporal coherence, and pixel-level associations over time. We present extensive quantitative and qualitative evaluations of both vision-based and LiDAR-based MOPT that demonstrate encouraging results.
  arXiv  Detail & Related papers  (2020-04-17T11:45:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.