MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding

• URL: http://arxiv.org/abs/2601.05495v1
• Date: Fri, 09 Jan 2026 02:59:05 GMT
• Title: MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding
• Authors: Zizhong Li, Haopeng Zhang, Jiawei Zhang,
• Abstract summary: Long videos, ranging from minutes to hours, present significant challenges for current Multi-modal Large Language Models.<n>We introduce MMViR, a novel multi-modal, multi-grained structured representation for long video understanding.<n> MMViR achieves a 19.67% improvement in hour-long video understanding while reducing processing latency to 45.4% of the original.
• Score: 7.663791315702202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long videos, ranging from minutes to hours, present significant challenges for current Multi-modal Large Language Models (MLLMs) due to their complex events, diverse scenes, and long-range dependencies. Direct encoding of such videos is computationally too expensive, while simple video-to-text conversion often results in redundant or fragmented content. To address these limitations, we introduce MMViR, a novel multi-modal, multi-grained structured representation for long video understanding. MMViR identifies key turning points to segment the video and constructs a three-level description that couples global narratives with fine-grained visual details. This design supports efficient query-based retrieval and generalizes well across various scenarios. Extensive evaluations across three tasks, including QA, summarization, and retrieval, show that MMViR outperforms the prior strongest method, achieving a 19.67% improvement in hour-long video understanding while reducing processing latency to 45.4% of the original.

Related papers

• Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models [28.68367581677484]
  We introduce a novel end-to-end schema for long-form video understanding, which includes an information-density-based adaptive video sampler (AVS) and an autoencoder-basedtemporal video compressor (SVC) integrated with a multimodal large language model (MLLM)<n>Our proposed system offers two major advantages: it adaptively captures essential information from video sequences of varying durations, and it achieves high compression rates while preserving crucial discriminative information.
  arXiv  Detail & Related papers  (2026-02-19T22:04:27Z)
• Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment [0.0]
  We propose UMaT, a framework that unifies visual and auditory inputs as structured text for large language models.<n>It significantly improves state-of-the-art Long Video Question Answering accuracy.
  arXiv  Detail & Related papers  (2025-03-12T05:28:24Z)
• SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis [52.050036778325094]
  We introduce SALOVA: Segment-Augmented Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content.<n>We present a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich context.<n>Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries.
  arXiv  Detail & Related papers  (2024-11-25T08:04:47Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)
• From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding [52.696422425058245]
  MultiModal Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks.<n>Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding.
  arXiv  Detail & Related papers  (2024-09-27T17:38:36Z)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding [41.9477837230283]
  LongVideoBench is a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes. We formulate a novel video question-answering task termed referring reasoning.
  arXiv  Detail & Related papers  (2024-07-22T16:00:55Z)
• V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning [65.30592368928723]
  Video summarization aims to create short, accurate, and cohesive summaries of longer videos.<n>Most existing datasets are created for video-to-video summarization.<n>Recent efforts have been made to expand from unimodal to multimodal video summarization.
  arXiv  Detail & Related papers  (2024-04-18T17:32:46Z)
• LongVLM: Efficient Long Video Understanding via Large Language Models [55.813206751150716]
  LongVLM is a simple yet powerful VideoLLM for long video understanding. We encode video representations that incorporate both local and global information. Our model produces more precise responses for long video understanding.
  arXiv  Detail & Related papers  (2024-04-04T11:33:29Z)
• Hierarchical3D Adapters for Long Video-to-text Summarization [79.01926022762093]
  multimodal information offers superior performance over more memory-heavy and fully fine-tuned textual summarization methods. Our experiments demonstrate that multimodal information offers superior performance over more memory-heavy and fully fine-tuned textual summarization methods.
  arXiv  Detail & Related papers  (2022-10-10T16:44:36Z)
• Deep Multimodal Feature Encoding for Video Ordering [34.27175264084648]
  We present a way to learn a compact multimodal feature representation that encodes all these modalities. Our model parameters are learned through a proxy task of inferring the temporal ordering of a set of unordered videos in a timeline. We analyze and evaluate the individual and joint modalities on three challenging tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action recognition.
  arXiv  Detail & Related papers  (2020-04-05T14:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.